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Overview 


The Children’s Institute, Inc. (Cll), is a multiservice organization in Los Angeles, California, that 
combines clinical mental health and other supportive services to meet the needs of children and their 
families who have been affected by trauma, such as physical or sexual abuse, domestic violence, or 
violence in the community. Through its Integrated Service Model, Cll provides holistic and coordi- 
nated support to children and families by potentially engaging them in multiple services: clinical 
services to address children’s mental health needs, programs for parents and guardians to help them 
better support their children, and youth activities to develop protective factors. The comprehensive 
nature of this model sets it apart from the often fragmented and uncoordinated child welfare system. 
A central aspect of Cll’s model is using evidence-based practices — highly specified treatment 
models that research has shown to be effective in treating a targeted population — in its clinical 
services. 

The Cll evaluation had two main components: an implementation study of Cll’s service model and a 
study of Cll’s delivery of evidence-based practices, including an in-depth fidelity study of its 
Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) services. 

Key Findings 

• Cll is achieving its goal of engaging clients in multiple services to holistically meet their needs. 
A majority of clients receiving clinical services from Cll also participated in another service at 
the organization. 

• Analysis of management information system data indicates that nearly a third of the children 
engaged in clinical services received an evidence-based practice. While little is known about 
national norms for the use of evidence-based practices, the study’s findings suggest that Cll is a 
leader in providing them. 

• Analysis also indicates that the dosage levels of Functional Family Therapy and TF-CBT — 
two prominent evidence-based practices at Cll — were both in line with model expectations. 

• The in-depth fidelity study of TF-CBT indicated that Cll’s implementation of the treatment 
model was aligned with that of other community-based organizations in similar fidelity stud- 
ies. The average client had at least a 50 percent chance of receiving half of the model’s core 
components. 

A Technical Resource for this report presents the complete set of findings from the in-depth fidelity 
study of Cll’s delivery of TF-CBT and is available on the MDRC website. 

Cll is also involved in MDRC’s Building Bridges and Bonds study of fatherhood programs, funded 
by the U.S. Department of Health and Human Services. 
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Preface 


There is overwhelming evidence that traumatic experiences in childhood — such as physical or 
sexual assault, gang violence, domestic violence, or sudden loss of a loved one — can lead to 
poor outcomes in adulthood. While the child welfare field is extensive and works to improve 
the life prospects of trauma-affected children and families, the available services are neverthe- 
less often fragmented and uncoordinated. Research has identified evidence-based practices that 
improve outcomes for these children and families, and there has been a push at the federal level 
in recent years to increase the use of such practices in children’s mental health care. However, 
many of the current services available lack evidence of their effectiveness. 

In this context, the Los Angeles-based Children’s Institute, Inc. (CII), operates its wide 
range of programs and services, including clinical mental health services, early child care and 
Head Start programs, programs for parents and guardians, and youth development activities. 
Integrating and coordinating these services to address the holistic needs of children and families 
is a critical component of CII’s service model, as is the use of evidence-based practices in 
mental health treatment when appropriate. 

This report describes how in implementing its Integrated Service Model CII sought to 
overcome the barriers associated with the fragmented and uncoordinated child welfare system 
through an approach that attempts to identify clients’ full range of needs and ensure they receive 
all the support required to address those needs. It offers lessons in how multiservice organiza- 
tions such as CII can structure services to meet the holistic needs of clients. Integrating services 
as CII has done, however, is not without its challenges. Tailoring services to the varied needs of 
each client requires navigating the complex funding system on which multiservice organizations 
rely, and which includes public agencies, private foundations, and health insurance providers. 

The report also adds to the understanding of the challenges of implementing evidence- 
based practices in community-based settings, where the highly specified protocols of these 
practices meet the realities of providing services in high-needs and under-resourced communi- 
ties. The in-depth fidelity study of CII’s delivery of Trauma-Focused Cognitive Behavioral 
Therapy services highlighted some of the difficulties therapists encounter when delivering a 
structured treatment to high-needs clients. The study found that therapists at CII did not provide 
all of the model’s required treatment components, which is consistent with findings from other, 
similar studies of community-based providers. This finding suggests the need for robust and 
low-cost tools to help providers deliver evidence-based treatments with fidelity. 

Gordon L. Berlin 

President, MDRC 
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Executive Summary 


Child abuse and neglect are significant problems in the United States, touching millions of lives 
each year. The U.S. Department of Health and Human Services reported that in fiscal year 
2012, 3.2 million children nationwide were the subject of a report to child protective services. 
The majority of youth in the child welfare system exhibit behavioral or social issues that are 
severe enough to warrant mental health treatment, a rate up to five times greater than mental 
health needs among their peers in the community who are not involved in the child welfare 
system. 1 


Recent trends in the children’s mental health care field have indicated that these behav- 
ioral and emotional issues can be largely attributed to trauma experienced earlier in life, which 
leads to “toxic stress responses” that can have a wide variety of adverse psychological and 
physiological consequences, some of which continue into adulthood. 2 Trauma can result from 
many experiences and events, including physical or sexual assault, gang violence, domestic 
violence, serious accidents, sudden or violent loss of a loved one, and natural disasters. 3 Chil- 
dren do not have to be the direct victims of violence to be affected by it; researchers have shown 
that exposure to community violence, such as hearing gun fire, has traumatic effects on chil- 
dren. 4 Child welfare organizations throughout the country combat trauma in all its fonns 
through a combination of prevention programs, direct services to affected families, and advoca- 
cy. While the lifelong impact of childhood mental illness and trauma is well documented, many 
children and youth do not receive the mental health treatment they need. Even when they do 
receive treatment, services may be inadequate or ineffective, and are often fragmented and 
uncoordinated. 


'This estimate is based on studies across child welfare systems in several states, as well as recent results of 
the National Survey for Child and Adolescent Well-Being. John A. Landsverk, Barbara J. Bums, Leyla F. 
Stambaugh, and Jennifer A. Rolls Reutz, Mental Health Care for Children and Adolescents in Foster Care: 
Review of Research Literature (Seattle, WA: Casey Family Programs, 2006). 

2 Wendy K. Silverman, Claudio D. Ortiz, Chockalingham Viswesvaran, Barbara J. Burns, David J. Kolko, 
Frank W. Putnam, and Lisa Amaya-Jackson, “Evidence-Based Psychosocial Treatments for Children and 
Adolescents Exposed to Traumatic Events" Journal of Clinical Child and Adolescent Psychology’ 37, 1 
(2008): 156-183; Vincent J. Felitti, Robert F. Anda, Dale Nordenberg, David F. Williamson, Alison M. Spitz, 
Valerie Edwards, Mary P. Koss, and James S. Marks, “Relationship of Childhood Abuse and Household 
Dysfunction to Many of the Leading Causes of Death in Adults: The Adverse Childhood Experiences (ACE) 
Study,” American Journal of Preventive Medicine 14, 4 (1998): 245-258. 

3 Child Welfare Committee, National Child Traumatic Stress Network, Child Welfare Trauma Training 
Tool Kit: Comprehensive Guide (2nd ed.) (Los Angeles and Durham, NC: National Center for Child Traumatic 
Stress, 2008). 

4 Joy D. Osofsky (ed.). Children in a Violent Society (New York: Guilford Press, 1998). 
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This report presents findings from a study of the Children’s Institute, Inc. (CII). A mul- 
tiservice organization in Los Angeles, CII combines a broad range of clinical and nonclinical 
services to meet the needs of children and families who have been affected by trauma. Each 
year, CII serves more than 20,000 children and family members. CII’s range of activities, which 
it calls its Integrated Service Model, serve the “whole child, entire family.” Through its service 
model, CII provides a broad range of supports that the child and family may need to overcome a 
history of abuse or trauma, including clinical services to address mental health needs, programs 
for parents and guardians to help them better support their children, and nonclinical youth 
development activities to help children and youth acquire protective factors. 5 CII also operates 
child care and Head Start programs for young children. CII’s treatment approach is trauma 
informed, and its services are designed to directly address the impact of trauma on children’s 
lives. An important aspect of CII’s Integrated Service Model is its focus on using evidence-based 
practices in clinical services. Evidence-based practices are highly specified treatment models that 
research has shown to be effective in treating specific symptoms in target populations. 

The report’s findings are based on work supported by the Social Innovation Fund (SIF), 
a program of the Corporation for National and Community Service. 6 The SIF combines public 
and private resources to increase the impact of innovative, community-based solutions with 
compelling evidence of improving the lives of people in low-income communities. As part of 
the Edna McConnell Clark Foundation SIF project, which focuses on children and youth ages 9 
to 24 years, CII sought to expand its youth services, including the youth development activities 
offered in the Central and South Los Angeles neighborhoods. 

Child Trauma and Treatment 

While the lifelong impact of childhood mental illness and trauma is well documented, many 
children and youth do not receive the mental health treatment they need. Research has shown 
that many young people in need of such care either do not receive services, or, when they do, 
receive services that are inadequate or ineffective and often unsupported by evidence. 7 This 
report uses the tenn “usual care” to describe mental health care that is not based on evidence. 8 


Protective factors are characteristics of individuals, families, or communities that mitigate risks to health 
and well-being. Examples include positive social connections, parenting skills and knowledge of child 
development, and effective communication practices. 

6 CI1 is one of 12 evidence-based programs selected in 201 1 to be part of the SIF program. EMCF matched 
$30 million from the SIF program with $30 million from its own endowment. The True North Fund, developed 
by EMCF in 201 1, helped the 12 SIF grantees secure the $60 million they were required by statute to raise to 
match this funding. 

7 Barbara J. Burns, E. Jane Costello, Adrian Angold, Dan Tweed, Dalene Stangl, Elizabeth M. Z. Farmer, 
and A1 Erkanli, “Children’s Mental Health Service Use across Service Sectors,” Health Affairs 14, 3 (1995): 
147-159; Sheryl H. Kataoka, Lily Zhang, and Kenneth B. Wells, “Unmet Need for Mental Health Care among 

(continued) 
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At the federal level, there has been a significant push to incorporate evidence-based 
practices into mental health care for children and youth. Unlike many other types of mental 
health care, the treatment models for evidence-based practices are well defined. Each model or 
treatment practice typically specifies the target population for which the treatment has been 
shown to be effective, as well as the treatment’s outcomes, content, dosage, and duration. While 
the current trend in the child welfare field is to increase the use of evidence-based practices, not 
every client may be appropriate to receive one of these treatments since they target highly 
specified symptoms or age groups. 

Providing empirically supported treatment in community-based settings presents a host 
of challenges. Community-based settings are often quite different from the university research 
settings where the practices are usually first developed and tested. Community-based providers 
typically serve more diverse and higher-risk populations and have larger caseloads. Whether or 
not an evidence-based treatment offered in a community-based setting is effective depends on 
how the provider implements it. In order to transfer efficacy from research to practice, providers 
must implement the treatment with fidelity to the model that was originally tested, which can be 
particularly challenging in community-based settings. Fidelity encompasses a number of areas: 
staff training practices, targeting the appropriate population, administering the correct dosage 
and frequency of treatment, and adherence to the prescribed model. 

Another challenge to effectively treating children and youth with mental health needs is 
that services are often fragmented and uncoordinated. Available services are often spread across 
different agencies, and funding streams support only specific types of care or treatment. This 
fragmentation limits the ability of providers to meet the full range of needs of children and 
families, which may include clinical mental health care, services to help the parents or guardi- 
ans better support their children, and child and youth development activities to help the children 
and youth acquire protective factors and succeed in school. 

Overview of CM 

CII operates in three of Los Angeles County’s eight Service Planning Areas: Downtown Los 
Angeles, Watts-South Central, and Torrance-Long Beach. Each of these areas is “high need,” 
which means that a significant proportion of adult residents are low income, have not completed 
high school, have poor physical and mental health, and have experienced or reported abuse or 
community violence. In these areas, CII strives to implement a neighborhood approach, where- 


US Children: Variation by Ethnicity and Insurance Status,’’ American Journal of Psychiatry 159, 9 (2002): 
1548-1555. 

8 The term “usual care” refers to treatments that are not empirically supported. Usual care is a term com- 
monly used in the medical field to refer to the treatment received by patients in the control group of a random- 
ized controlled trial. 
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by CII builds relationships with residents and institutions, such as schools, churches, and other 
child welfare agencies. 

CII’s services are roughly divided into four programmatic categories: clinical mental 
health services, family support, child and youth development, and early childhood care and 
education. 

• Clinical mental health services include diagnosis and assessments of mental 
health needs, individual and group therapy, and family therapy. Licensed 
therapists or psychologists typically deliver these services, which may in- 
clude evidence-based or evidence-informed practices. 9 

• Family support includes programs offered to parents or guardians. These 
programs address parent education, child development, and family economic 
success and stability through case management, parenting classes, support 
groups for fathers and grandparents, financial literacy workshops, and job- 
readiness supports. 

• Youth development includes nonclinical activities, such as programs for 
young people of different ages that address life skills, social skills, literacy 
and education, creative arts, and health and wellness. 

• Early childhood care and education services are for infants and children 
from birth to 5 years of age. They include Head Start and child care pro- 
grams. Though early childhood programming encompasses more than one- 
fifth of CII’s overall budget, these services were not the focus of this SIF ini- 
tiative, which targeted youth ages 9 to 24 years. 

As an operating philosophy, CII coordinates the services it provides to meet the holistic 
needs of children and their families. This approach stands in contrast to the fragmented services 
that often characterize the child welfare system. CII conceptualized this philosophy around three 
components: recovery, resiliency, and readiness. 

• Recovery from adverse childhood experiences involves reducing the effects 
of trauma and high-risk behaviors. Recovery is the primary focus of CII’s 
clinical services. 

• Resiliency is the capacity of young people and their families to persevere 
and prevent the effects of trauma, and it is developed by enhancing protective 


9 Evidence-informed practices are treatments that share characteristics with evidence-based practices but 
fall short of the required threshold of evidence. 
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factors and reducing risks. It is the primary focus of early childhood, family 
support, and youth development programming. 

• Readiness for success in school, work, and life involves positive and healthy 
personal behaviors and social relationships, engagement with education or 
occupational training, and the ability to connect to supports or resources. 

CII’s combined services support readiness. 

Through its Integrated Service Model, CII knits these components together to address 
the complex needs of the families it serves. Depending on their needs, clients may receive 
multiple types of services throughout their involvement with CII. The Integrated Service Model 
aims not to simply offer multiple services but to eliminate operating silos among its various 
services and create a system that accurately identifies clients’ full range of needs and ensures 
they receive all the support required to address those needs. This report in large part assesses 
CII’s implementation of the Integrated Service Model. 

The CII Evaluation 

Building evidence is a core component of the SIF, and each SIF grantee is required to undergo 
an evaluation of its service model. The evaluation of CII consisted of two main components: an 
implementation study of CII’s service model and a study of CII’s delivery of evidence-based 
practices, including an in-depth fidelity study of its delivery of Trauma-Focused Cognitive 
Behavioral Therapy (TF-CBT). Both components focused on understanding how the agency 
implements services to address issues of childhood trauma. Overall, this evaluation provided 
CII staff with an independent review of its services, delivery model, and data system. Further- 
more, it serves as a case study for other program operators or policymakers in how to structure 
services to meet the holistic needs of clients and how to overcome the barriers to effectively 
serving children associated with a fragmented child welfare system. 

The study attempted to answer three main questions: 

• How do CII’s services and delivery model meet the needs of the diverse 
population it serves, particularly services provided to children and youth 9 to 
24 years old? 

• How does CII’s Integrated Service Model combine and coordinate its clinical 
and nonclinical services to address the holistic needs of children and fami- 
lies? 

• How does CII integrate evidence-based practices, particularly Functional 
Family Therapy (FFT) and TF-CBT, into its array of clinical services? 
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To answer these questions, the MDRC research team analyzed a mix of quantitative and 
qualitative data. The quantitative data included clients’ demographic data and service participa- 
tion records for services received during 2012 and 2013; the research team collected these data 
from CII’s management information system. The MDRC research team also gathered qualita- 
tive data about program operations through interviews with CII staff and representatives from 
some of its partners, primarily during three site visits in 2013. The team also asked CII clini- 
cians and their supervisors to complete a web-based survey in 2013. In partnership with MDRC, 
a team from the Medical University of South Carolina conducted a fidelity study of CII’s 
delivery of TF-CBT services using an observational method. 

Implementation of CII Services 

Analyses of these data point to the following findings: 

• CII’s Integrated Service Model is innovative and highly ambitious. 

CII’s Integrated Service Model seeks to overcome the shortcomings of the child welfare 
system, but CII staff still had to work within that fragmented system to fund its services. CII 
staff confronted numerous external funding constraints to providing services in the holistic way 
envisioned by the Integrated Service Model. CII funds its programs and services through a 
combination of service fees and contracts. However, each of these funding streams comes with 
a host of requirements and stipulations. As a result, the set of services that a client may need 
does not always fit neatly into one of the available funding streams. To tailor services to each 
client’s needs, CII staff must therefore find flexible funding streams or creatively combine 
contractual or other funding streams. 

• Preliminary analysis shows that CII’s implementation of the Integrated 
Service Model appears to be strong with respect to clients receiving clin- 
ical services. 

CII strives to provide multiple services to clients to address their many needs and high- 
risk factors. The overwhelming majority of clients receiving CII’s clinical services also partici- 
pated in another service at CII, and nearly half of clients in clinical services participated in all 
three types of services. It was not possible to fully assess CII’s progress in implementing the 
Integrated Service Model because the research team did not have access to data about clients’ 
risk factors, which are integral to determining whether clients had umnet needs. 
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• CII is a leader in adopting and implementing evidence-based practices; 
nearly a third of clients receiving clinical services engaged in an evi- 
dence-based practice . 10 

CII also provides a number of evidence-informed therapies; more than 20 percent of 
clients in the analysis received one of these treatments. 11 CII is viewed as a leader in implement- 
ing and providing evidence-based practices. As an early adopter, CII first began incorporating 
evidence-based practices into its clinical model in 1999. Little is known about national nonns 
for usage of evidence-based practices; according to one estimate of youth receiving care 
through California’s county mental health plans, only 2 percent of youth received an evidence- 
based practice. 12 Importantly, evidence-based practices are neither appropriate for every client 
nor are they available for every age group. While the proportion of CII’s clients receiving 
evidence-based practices exceeds the California estimate many times over, it is difficult to know 
whether this concentration of clients receiving these practices was appropriate without knowing 
more about each client’s circumstances. This study, however, was not designed for such an 
analysis, and the research team did not have the detailed data about each client to conduct one. 

• The dosage levels of both FFT and TF-CBT aligned with model expecta- 
tions. 

Analysis of data from CII’s management infomiation system indicated that, on average, 
clients receiving FFT attended 15 sessions over the course of five months. Similarly, clients 
receiving TF-CBT on average attended 19 sessions over five months. The dosage of both 
therapies fell within the bounds set by their respective treatment models, although both fell on 
the higher end of the models’ acceptable ranges. 13 Data were not available on factors that may 
have contributed to the relatively high number of sessions; however, CII staff indicated in 
interviews that the complex nature of the clients’ history of trauma and tumultuous lives often 
resulted in frequent family crises during treatment. 


10 Evidence-based practices include the following treatments and programs: Cognitive Behavioral Inter- 
vention for Trauma in Schools, Child Parent Psychotherapy, FFT, Incredible Years, Managing and Adapting 
Practice, Multidimensional Treatment Foster Care, Parent-Child Interactive Therapy, Reflective Parenting 
Program, TF-CBT, and Trauma Systems Therapy-Substance Abuse. 

1 'Evidence-informed practices include the following treatments and programs: Domestic Violence Treat- 
ment Groups, Project Fatherhood, Wraparound services, Youth with Sexual Behavior Problems, and social 
skills and parent support groups. These practices are informed by some evidence but not as much as evidence- 
based practices have accumulated. 

12 Technical Assistance Collaborative and Fluman Services Research Institute, California Mental Health 
and Substance Use System Needs Assessment Appendices (California Department of Flealth Care Services, 
2012). Website: www.dhcs.ca.gov/provgovpart/Documents/Data%20Appendices%203%201%2012.pdf. 

l3 The acceptable range for FFT is 8 to 12 sessions; the acceptable range for TF-CBT is 12 to 20 sessions. 
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• CII’s fidelity to the TF-CBT model was in line with previous fidelity 
studies of TF-CBT in community-based settings. Clients on average 
were more than 50 percent likely to receive half of the treatment’s core 
components. 

A rigorous examination of the adherence of therapists at CII to the TF-CBT model us- 
ing an observational method found that the average client had at least a 50 percent chance of 
receiving half of the core components of the model. 14 The finding that clients did not receive all 
components of TF-CBT is consistent with other research on the implementation of TF-CBT in 
community-based settings. Clients were most likely to receive the cognitive coping, relaxation, 
affective expression and modulation, psychoeducation, and trauma narrative components of TF- 
CBT. The study found therapists delivered the parent component at low rates, which is also 
consistent with prior research. 15 The study found that fidelity varied at the client level rather 
than the therapist level, indicating that clients seen by the same therapist could have had varying 
experiences with TF-CBT. 

The fidelity study also found that a therapist self-report tool, the Brief Practice Check- 
list, led to similar conclusions about the usage of TF-CBT components as did an observational 
method. This finding indicates that the Brief Practice Checklist may be a promising low-cost 
tool to monitor fidelity. Observational methods of monitoring fidelity, such as the one used in 
this study, are time and resource intensive and not practical on a large scale for many communi- 
ty-based organizations. Therapists and supervisors could use the Brief Practice Checklist to 
monitor whether or not therapists are delivering the TF-CBT components, and supervisors 
could use the information in the checklist to advise therapists on cases and on how to eliminate 
any roadblocks to providing the treatment as intended. Organizations could also use data from 
the checklists to compare differing outcomes among cases and identify and assess any patterns. 
However, there are some limitations to using the checklist on its own to evaluate fidelity. 
Therapists in this study had the tendency to over-report their use of components, relative to the 
observational data. Additionally, observational methods can measure the extent to which 
therapists implement each component, whereas the Brief Practice Checklist can only measure 
whether or not therapists implement the components. However, organizations could use the 
Brief Practice Checklist in combination with others tools, such as periodic direct or audio- 
recorded observations, to monitor fidelity more comprehensively. The use of the Brief Practice 
Checklist as a fidelity tool merits further study. 


14 A Technical Resource for this report presents the full study and is available on the MDRC website at 
www.mdrc.org. 

15 TF-CBT requires that the therapist meet separately with the child and the parent or guardian with similar 
frequency, and meet conjointly with both at particular points during treatment. 
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Conclusion 


As policymakers, practitioners, and researchers in the child welfare field work to improve 
services available through the child welfare system, CII and its experience developing and 
implementing its Integrated Service Model as well as delivering evidence-based practices offer 
important lessons. These lessons could be useful not only to similar multiservice organizations 
but to all those in the child welfare field looking for the best ways to serve children through an 
often fragmented child welfare system. Those interested in evidence-based practices may find 
the findings from the fidelity study of TF-CBT useful. These findings suggest that one area for 
further research could be investigating how to cost-effectively combine self-reporting tools and 
observational methods to support fidelity. 
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Chapter 1 

Introduction 


Child abuse and neglect are significant problems in the United States, touching millions of lives 
each year. Traumatic events in childhood can have effects throughout an individual’s lifespan, 
diminishing the chances for a happy, productive life. Child welfare organizations throughout the 
country combat these issues through a combination of prevention programs, direct services to 
affected families, and advocacy. 

This report describes how the Children’s Institute, Inc. (CII), a multiservice organiza- 
tion in Los Angeles, combines mental health care and other supportive services to address the 
issue of childhood trauma. Each year, CII serves more than 20,000 children and their families, 1 
providing them an array of services through its Integrated Service Model. Developed to serve 
the “whole child, entire family,” the model aims to identify, coordinate, and deliver a broad 
range of supports that children and their families may need to overcome a history of abuse or 
trauma. These supports include early childhood care and Head Start programs, clinical mental 
health services, programs for parents and guardians to help them better support their children, 
and youth development activities to help children and youth develop protective factors. 2 CII’s 
treatment model is trauma informed and designed to directly address the impact of trauma on 
children’s lives. A central aspect of the model is its focus on using evidence-based practices in 
clinical services. Evidence-based practices are highly specified treatment models that research 
has shown to be effective in treating a target population. 

In 2010, CII received a Social Innovation Fund (SIF) grant through the Edna 
McConnell Clark Foundation (EMCF) to expand its services, including the youth development 
programs in the Central and South Los Angeles neighborhoods. The initiative focuses on youth 
who are ages 9 to 24 years. (See Box 1.1 for more information on the EMCF SIF initiative.) 

Building evidence is a core component of the SIF, and each SIF grantee is required to 
undergo an evaluation of its service model. The evaluation of CII centered on two main ele- 
ments: an implementation study of CII’s service model and an in-depth fidelity study of CII’s 
delivery of Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) services. 


'Children’s Institute, Inc. (2014). 

Protective factors are characteristics of individuals, families, or communities that mitigate risks to health 
and well-being. Examples include positive social connections, parenting skills and knowledge of child 
development, and effective communication practices. 
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Box 1.1 


The EMCF SIF Initiative 

The Social Innovation Fund (SIF), an initiative enacted under the Edward M. Kennedy Serve 
America Act, directs millions of dollars in public-private funds to expand effective solutions 
across three issue areas: economic opportunity, healthy futures, and youth development and 
school support. This work seeks to create a catalog of proven approaches that can be replicated 
in communities across the country. The SIF assembles a three-to-one private-public funding 
match, sets a high standard for evidence, empowers communities to identify and drive solu- 
tions to address social problems, and creates an incentive for grant-making organizations to 
target funding to promising programs more effectively. Administered by the Corporation for 
National and Community Service (CNCS), the SIF is part of the federal government’s broader 
agenda to redefine how evidence, innovation, service, and public-private cooperation can be 
used to tackle urgent social challenges. 

The Edna McConnell Clark Foundation (EMCF), in collaboration with MDRC and The 
Bridgespan Group, is leading a SIF initiative that aims to enlarge the pool of organizations 
with programs proven to help low-income young people make the transition to productive 
adulthood. The initiative particularly focuses on young people who are at greatest risk of 
failing or dropping out of school or not finding work, who are involved or likely to become 
involved in the foster care or juvenile justice system, who engage in risky behavior such as 
criminal activity, or who experience teenage pregnancy. 

EMCF and its partners selected a first cohort of nine organizations and a second cohort of three 
to receive SIF grants: BELL (Building Educated Leaders for Life), Center for Employment 
Opportunities, Children’s Aid Society-Carrera Adolescent Pregnancy Prevention Program, 
Children’s Home Society of North Carolina, Communities in Schools, Gateway to College 
Network, PACE Center for Girls, Reading Partners, The SEED Foundation, WINGS for Kids, 
Youth Guidance, and Children’s Institute, Inc. These organizations were selected through a 
competitive selection process based on prior evidence of impacts on economically disadvan- 
taged young people, a track record of serving young people in communities of need, strong 
leadership and a potential for growth, and the financial and operational capabilities necessary 
to expand to a large scale. 

The EMCF SIF grant, called the True North Fund, includes support from CNCS and 15 private 
co-investors: EMCF, The Annie E. Casey Foundation, The Duke Endowment, The William 
and Flora Hewlett Foundation, The JPB Foundation, George Kaiser Family Foundation, The 
Kresge Foundation, Open Society Foundations, The Penzance Foundation, The Samberg 
Family Foundation, The Charles and Lynn Schusterman Family Foundation, The Starr Foun- 
dation, Tipping Point Community, The Wallace Foundation, and the Weingart Foundation. 
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The implementation study sought to answer three principal questions: 

• How does CII’s program model and services meet the needs of the diverse 
population it serves, particularly those of children and youth ages 9 to 24 
years? 

• How does CII’s Integrated Service Model combine clinical services and non- 
clinical services to address the holistic needs of children and families? 

• How does CII integrate evidence-based practices, particularly Functional 
Family Therapy (FFT) and TF-CBT, into its various clinical services? 

The fidelity study assessed CII’s delivery of TF-CBT and the value of a therapist self- 
report tool in monitoring fidelity. 

The key findings of the study were: 3 

• CII’s Integrated Service Model is innovative and highly ambitious, as it 
seeks to overcome the shortcomings of how child welfare organizations typi- 
cally operate. CII confronts numerous funding constraints that it must work 
around to provide services in the holistic way envisioned by the model. 

• Preliminary analysis shows that the Integrated Service Model is well imple- 
mented in CII’s clinical services. The overwhelming majority of clients re- 
ceiving clinical services also participated in another, nonclinical service at 
CII. Nearly half of clients in clinical services also participated in parent or 
guardian programs and youth development activities. 

• CII is a leader in adopting and delivering evidence-based practices, and near- 
ly a third of clients engaged in clinical services received an evidence-based 
treatment. However, it is difficult to interpret the saturation of clients receiv- 
ing evidence-based treatments without more information about each client’s 
circumstances and suitability for such treatments. 

• Analysis of data from CII’s management information system records indicat- 
ed that fidelity to FFT and TF-CBT dosage aligned with the respective mod- 
els’ prescribed dosages. 


3 Data from CII’s management information system were analyzed for clients who enrolled between Janu- 
ary 1, 2012, and June 30, 2013, and received services January 1, 2012, through December 31, 2013. Interviews 
with CII staff took place over three site visits in 2013. 
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• The fidelity study found that on average clients were more than 50 percent 
likely to receive half of the core components of TF-CBT, which is in line with 
findings from previous fidelity studies of TF-CBT delivered in community- 
based settings. 


Child Trauma and Treatment 

CIFs mission is to help children overcome the adverse effects of trauma on their lives. This 
section provides some background on the issue of childhood trauma, and describes how the 
child welfare system attempts to address the issue and how national and local policies shape the 
services provided. 

The Effects of Childhood Trauma 

The country’s child welfare system confronts serious and widespread threats to the safe- 
ty of children. The U.S. Department of Health and Human Services reported that in federal 
fiscal year 2012, 3.2 million children were the subject of a report to child protective services. Of 
these reports, 678,810 were substantiated. As the most populous state, it is not surprising that 
California had the largest share of child abuse victims, 76,026, or 1 1 percent of the nation’s total 
substantiated victims. 4 

The impact of trauma on children is well documented. Between half and three-fourths 
of youth in the child welfare system exhibit behavioral or social issues serious enough to 
warrant mental health treatment, a rate up to five times greater than that among youth who are 
not involved in the child welfare system. 5 Recent trends in the children’s mental health care 
field have indicated that these behavioral and emotional issues can be largely attributed to 
trauma experienced earlier in life. 6 Trauma can result from many experiences and events, 
including physical or sexual assault, gang violence, domestic violence, serious accidents, 
sudden or violent loss of a loved one, and natural disasters. 7 Children do not have to be the 
direct victim of violence to be affected by it; research has shown that exposure to community 
violence has traumatic effects on children. 8 In a national survey of adolescents ages 12 to 17 
years, nearly 40 percent reported witnessing violence, and close to 25 percent reported being 


4 U.S. Department of Health and Human Services, Administration for Children and Families, Administra- 
tion on Children, Youth and Families, Children’s Bureau (2013). 

5 This estimate is based on studies across child welfare systems in several states, as well as recent results of 
the National Survey for Child and Adolescent Well-Being. See Landsverk, Burns, Stambaugh, and Rolls Reutz 
(2006). 

6 Silverman et al. (2008); Felitti et al. (1998). 

7 Child Welfare Committee, National Child Traumatic Stress Network (2008). 

8 Osofsky (1998). 
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physically assaulted. 9 Children and youth in the child welfare system are particularly at risk and 
have often experienced multiple traumatic events over time, with the event that precipitated a 
report to child protective services generally being the last in a long series. 

Studies have demonstrated that exposure to traumatic events has many adverse short- 
and long-term effects on children and youth. Psychologically, children and youth referred to 
child welfare experience high rates of post-traumatic stress disorder, with estimates ranging 
from 1 1 to 60 percent. 10 These youth also exhibit high rates of depression, anxiety, and attention 
deficit hyperactivity disorder. 1 1 These psychological and emotional responses often manifest as 
antisocial or otherwise maladaptive behaviors, which may in turn create more issues for them in 
school or in their foster families. 12 

Experiencing traumatic events as a child can also have negative effects on physical 
health. 13 The continual “toxic stress response” caused by traumatic events, particularly early in 
life, can have detrimental effects on brain development and may cause learning difficulties 
(which, in turn, can lead to negative psychological effects such as low self-esteem). 14 Substance 
abuse is another major health problem affecting this population; youth and adults who experi- 
enced childhood trauma are more likely to develop substance use disorders. 15 Adults who 
experienced multiple traumatic experiences in childhood are at significantly higher risk of 
developing substance use disorders, depression, and obesity; smoking tobacco; and contracting 
sexually transmitted diseases. 16 

Children living in poverty are particularly at risk. They are more likely to experience 
trauma and to experience multiple traumas over the course of their childhood than their more 
affluent peers. They are also less likely to have access to resources that address the negative 
effects of trauma. Their parents, who are likely experiencing stress from living in poverty as 
well as trauma of their own, are less likely to be effective in helping their children with their 
trauma-related challenges. 17 

CII’s Integrated Service Model is designed to address both the short- and long-term ef- 
fects of trauma on children and their families. As discussed in more detail in Chapter 5, CII 


9 Kilpatrick and Saunders (1999). 

10 Kolko et al. (2010); Dubner and Motta (1999); Ethier, Lemelin, and Lacharite (2004); Marsenich (2002). 
1 'Ackerman et al. (1998); Hildyard and Wolfe (2002); Marsenich (2002); Weisz and Gray (2008). 
12 Richardson, Henry, Black-Pond, and Sloane (2008); Child Welfare Committee, NCTSN (2008). 

13 Felitti et al. (1998). 

14 Center on the Developing Child at Harvard University (2010). 

15 Felitti et al. (1998); Keller, Salazar, and Courtney (2010); Landsverk, Bums, Stambaugh, and Rolls 
Reutz (2006). 

16 Felitti et al. (1998). 

17 Collins et al. (2010). 
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provides services directed at helping children recover from trauma (recovery), build protective 
factors such as coping strategies to prevent future trauma (resiliency), and prepare for produc- 
tive lives (readiness). 

Evidence-Based Treatment Practices 

While the lifelong impact of childhood mental illness and trauma is well documented, 
many children and youth do not receive the mental health treatment they need. Research has 
shown that many young people in need of mental health care either do not receive services, or, 
when they do, receive services that are inadequate or ineffective . 18 Another challenge to 
effective treatment is that children often receive fragmented and uncoordinated services because 
providers are spread across agencies and funding streams support only specific types of care or 
treatment. 

The mental health treatment that trauma-affected children receive is often not empirical- 
ly supported. This report uses the term “usual care” to describe mental health care that is not 
evidence based . 19 Therapists have diverse training backgrounds and therapeutic approaches, and 
thus what usual care typically entails can vary widely; there is also little research on usual care, 
which makes it challenging to categorize and assess the various types of treatment it typically 
encompasses. Therapists often mix and match approaches to tailor treatment to the individual. 
One example of usual care is psychodynamic therapy, which focuses on helping clients uncover 
the unconscious content of their psyche that underlies their issues. Another example is play 
therapy, which uses play to help children process difficult life experiences. In contrast, an 
evidence-based treatment is prescribed; while therapists may have slightly different clinical 
approaches, they administer the same treatment to any client who is prescribed it. 

At the federal level, there is growing momentum to increase the use of evidence-based 
practices in children’s mental health care. Evidence-based practices are those that research has 
shown to be effective. An evidence-based treatment model is well specified, defining the target 
population for which it has been proven to be effective, the intended outcomes, and its content, 
dosage, and duration. However, as such, a given evidence-based practice may not be appropri- 
ate for every client since the target population that benefits from it may be a subgroup of clients, 
for example, a particular age group or a group with specific symptoms. 

While evidence-based practices are gaining ground in the child welfare field, there is 
still a lack of consensus about what criteria should be used to determine whether a treatment is 


s Burns et al. (1995); Kataoka, Zhang, and Wells (2002). 

19 The term “usual care” in this report refers to treatments that are not empirically supported. Usual care is 
a term commonly used in the medical field to refer to the treatment received by patients in the control group of 
a randomized controlled trial. 
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evidence based. For the purposes of this report, the research team used the criteria that 
Chambless and Hollon delineate in their 1998 study. 20 Among other criteria, Chambless and 
Hollon set the threshold of evidence for such a practice at two randomized control trials, 
conducted by independent researchers. 21 In this report, the term “evidence-informed practice” 
refers to a practice that shares characteristics with evidence-based practices but falls short of this 
threshold of evidence. 

Fidelity is a major challenge to implementing evidence-based practices. Whether or not 
an evidence-based treatment is effective largely depends on how well service providers imple- 
ment it. In order to “transfer” efficacy from research to practice, providers must implement the 
treatment with fidelity to the model that was originally tested and proven to be effective. 
Evidence-based practices typically include an implementation plan, or a set of instructions. 22 
This plan may include guidelines for recruiting and training staff, monitoring the implementa- 
tion, and evaluating clients. Fidelity comprises several elements. One way to assess fidelity is to 
compare the implementation plan for a specific treatment with a provider’s actual implementa- 
tion. For example, researchers may compare the planned number of hours of staff training with 
the actual numbers of training that staff received. Another way to assess fidelity is to examine 
treatment fidelity, or the extent to which the treatment received by the client matches the 
intended treatment. For example, if the treatment called for a client to receive a curriculum 
consisting of five units, an evaluation of treatment fidelity would assess the extent to which the 
client received those five units as planned. 

Providing evidence-based practices with fidelity presents several challenges. The prac- 
tices require that providers adhere to the prescribed treatment methods, which may be a difficult 
adjustment for a provider accustomed to a different approach. Community-based settings, such 
as those where CII operates, pose particular challenges since they are often quite different from 
the university research settings where the practices are usually first developed and tested. 
Community-based providers typically serve more diverse and higher-risk populations and 
manage larger caseloads. 

Regarded by many as an early adopter, CII has embraced evidence-based and evidence- 
informed practices and has been integrating them into its clinical services since 1999. For this 
reason, this report focuses in part on CII’s implementation of evidence-based practices in its 
clinical services. However, since trauma-affected children and families often need other 


20 Chambless and Hollon (1998). 

2 'Chambless and Hollon (1998). Other entities use more relaxed criteria. Blueprints for Healthy Youth 
Development’s criteria for model programs requires two randomized controlled trials or one randomized 
controlled trial and one quasi-experimental design. See Blueprints for Healthy Youth Development (2015). 
22 Weiss, Bloom, and Brock (2013). 
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services, such as case management and coaching for parents or guardians, the report also 
assesses CII’s additional supports. 

CII takes a holistic approach to health care, combining mental health treatment with 
wraparound supports for families to meet the needs of the “whole child, entire family.” Called 
the Integrated Service Model, the approach includes clinical mental health services, programs 
for parents and guardians to help them better support their children, and youth activities to help 
children and youth develop protective factors. 

The Role of National and Local Policies 

The national and local policy context plays a significant role in how providers serve 
children and families in the child welfare system. Policies at the federal, state, and local levels 
can act as both facilitators and barriers to the successful implementation of interventions by 
organizations directly serving children. 

At the federal level, government agencies and nongovernmental entities help shape pol- 
icy related to the treatment and services children and families in the child welfare system 
receive and how those services are delivered and funded. National initiatives are responsible for 
the development of many of the evidence-based practices discussed in this report. 

The policy context in California, including laws and policies at the state and county 
levels, also has a major impact on how mental health services are funded and delivered. State 
policy constitutes much of the structure for how counties provide child welfare and mental 
health services. Counties also enact their own specific requirements and programs, and 
providers such as CII must navigate this complex system. Chapter 2 discusses these issues in 
more detail. 


Overview of This Report 

Building evidence is a core component of the SIF, and each SIF grantee is required to undergo 
an evaluation of its service model. Early discussions of the evaluation design for CII centered 
on conducting a randomized controlled trial. However, a randomized controlled trial proved to 
be untenable because it would have meant withholding treatment from children. Moreover, the 
research team determined that it was not necessary to conduct an experimental evaluation since 
a significant portion of CII’s clients receive evidence-based treatments that research has already 
shown to be effective. Accordingly, the premise of the evaluation design is that if CII is imple- 
menting the evidence-based treatments with fidelity, then it can be reasonably assumed that 
these treatments are achieving the intended results. The evaluation design thus called for in- 
depth fidelity studies of CII’s evidence-based practices. 



The evaluation design initially called for fidelity studies of CII’s three evidence-based 
practices, which according to available data nearly 50 percent of CII’s clients 9 to 24 years old 
were receiving. However, during the early stages of the evaluation, the research team decided to 
drop the fidelity study of the Cognitive Behavioral Intervention for Trauma in Schools because 
of small sample sizes and scaled back the study of FFT. Chapter 5 describes the remaining 
fidelity study of Trauma-Focused Cognitive Behavioral Therapy (TF-CBT). The in-depth 
fidelity study of TF-CBT assessed (1) the degree to which CII implemented TF-CBT with 
fidelity to the treatment model (treatment fidelity), (2) how fidelity might vary by client or 
therapist characteristics and (3) the utility of the TF-CBT Brief Practice Checklist in monitoring 
therapist fidelity. 23 The MDRC research team managed the data collection, and a team from the 
Medical University of South Carolina oversaw the data analysis. 

The evaluation design also included a broader implementation study of CII’s service 
model to provide context for the fidelity studies. Chapter 4 presents the findings from the 
implementation study. The implementation study set out to (1) describe how CII’s services 
model and services meet the needs of the diverse population it serves, with a focus on services 
provided to youth 9 to 24 years old (2) describe how CII’s Integrated Service Model combines 
clinical and nonclinical services to address the holistic needs of children and families, and 
(3) evaluate how CII implements evidence-based practices in its clinical services. 


Data Collection and Data Sources for the Implementation Study 

The MDRC research team collected various fonns of data for the implementation study cover- 
ing the years 2012 through 2014. 24 The team analyzed the data in 2014 and 2015. 

Data sources for the implementation study include: 

• Three site visits to CII in 2013, during which the research team conducted 
semi-structured interviews with 83 CII staff, including executive staff, clini- 
cians, and supervisors. 25 MDRC researchers also conducted semi-structured 
interviews with staff from eight other agencies in Los Angeles, including the 
Los Angeles County Department of Mental Health and providers serving 
similar populations. 


23 The TF-CBT Brief Practice Checklist is a self-monitoring tool for therapists to assess their fidelity to the 
treatment model. 

24 The research team secured institutional review board (IRB) approval from MDRC and CII to conduct 
interviews with CII staff. All staff who participated in interviews did so only after consenting. 

^Visits occurred in February, May, and October of 2013. 
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• A web-based survey of CII clinicians and their supervisors conducted in 
2013. The research team fielded the survey to 69 therapists and psychologists 
and 27 supervisors. The response rate was 39 percent. 

• Documents provided by CII, including assessments and outcome measures 
and promotional materials. 

• Records from CII’s management information systems, which included sepa- 
rate databases for clinical records, family support and youth development ac- 
tivities, outcome data, and staff records. The research team analyzed the data 
for services delivered in the calendar years 2012 and 2013. 26 

Appendix A describes the data sources for the implementation study, and some of their 
limitations, in more detail. 


Data Collection and Data Sources for the Fidelity Study 

Research subjects in the fidelity study included new TF-CBT clients and their assigned thera- 
pists. 27 Enrollment for the fidelity study began in November 2013 and ended in August 2014. A 
total of 126 TF-CBT clients and 34 therapists enrolled in the study. 

Data sources for the fidelity study include: 

• Demographic data on 126 clients enrolled in the study 

• Demographic, education, and caseload data on 34 therapists enrolled in the 
study 

• 108 TF-CBT Brief Practice Checklists 

• 1 ,009 audio recordings of client therapy sessions 


26 ln 2014, CII replaced the management information system from which the research team collected data 
for this report. CII expects the system, which incorporates new checks and balances to improve data quality, to 
alleviate many challenges. 

~ 7 The research team obtained 1RB approval to conduct the research from MDRC, CII, and the Medical 
University of South Carolina. The team secured additional approval from the Los Angeles County Juvenile 
Court and Department of Mental Health Human Subjects Review Committee. 
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Roadmap to the Report 

Chapter 2 provides an overview of CII’s organizational structure, including its history in the 
communities it serves, operating philosophy, and funding mechanisms. It explains recent SIF 
involvement and business planning that resulted in strategic changes. 

Chapter 3 describes the target population that CII serves, with an emphasis on 9- to 24- 
year-olds, and the pathways through which clients enter CII services. It also explains the client 
referral, assessment, and enrollment processes. 

Chapter 4 describes the implementation of CII services, including the staffing structure. 
It also explains the Integrated Service Model in detail. 

Chapter 5 examines CII’s implementation of evidence-based practices, including TF- 
CBT and FFT. The chapter assesses CII’s implementation of these evidence-based practices, 
and describes some barriers and facilitators to implementation. The chapter summarizes the 
findings from the TF-CBT fidelity study. (A Technical Resource for this report presents the full 
study and is available on the MDRC website.) 

Chapter 6 summarizes the report’s conclusions and identifies areas for further research. 
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Chapter 2 

Overview of Cl I 


The Children’s Institute, Inc. (CII), is a large organization with an operating budget of $50 
million, serving more than 20,000 children and family members each year at three multiuse 
campuses and dozens of centers and child care facilities throughout Los Angeles. This chapter 
begins with a summary of CII’s long and varied history, followed by a brief description of the 
communities it currently serves. The chapter then gives an overview of CII’s organizational 
structure, including the main services it provides, its operating philosophy and Integrated 
Service Model, and funding sources. It concludes with a description of how CII incorporates 
evidence-based practices into its clinical services. 


History and Background of CII 

Founded in Los Angeles in 1906 and originally named Big Sister League, CII began as a 
home for unwed pregnant women. The organization shifted the focus of services to child care 
in the 1960s, as more of the pregnant single women it served chose to keep their children 
rather than put them up for adoption. In the 1970s, CII started providing services to at-risk 
children in response to the growing need for child abuse prevention services. The organiza- 
tion changed its name to the Children’s Institute International in 1980 and continued to 
expand its services for at-risk children, including mental health services, in subsequent 
decades to become the large multiservice organization it is today. In 2006, it changed its 
name, again, to Children’s Institute, Inc. 


The Communities CII Serves 

CII operates in three of Los Angeles County’s eight Service Planning Areas (SPAs): SPA 4 
(Metropolitan Los Angeles), SPA 6 (South), and SPA 8 (South Bay). Figure 2.1 shows the 
SPAs that CII serves. In each SPA, CII operates multiple office locations. In these communities, 
CII strives to implement a neighborhood approach, whereby CII builds on connections with 
residents and other institutions in the community, such as schools, churches, and other child 
welfare agencies. While all the communities CII serves are considered high need, there are 
nonetheless regional variations among them with respect to socioeconomic demographics. (See 
Table 2.1.) 

CII has the largest and most significant presence in SPA 4, where its flagship Otis 
Booth campus opened in 201 1. The Otis Booth campus serves as CII’s headquarters; in addition 
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Figure 2.1 

Map of Los Angeles County Service Planning Areas Served by CII 



Service Planning Areas 

1 - Antelope Valley 

2 - San Fernando 

3 - San Gabriel 

4 - Metro 

5 - West 

6 - South 

7 - East 

8 - South Bay 

Shaded regions served by CII 


to extensive client services, the location houses many of CII’s managerial functions, including 
its human resources, training, billing, and research and evaluation departments. SPA 4 is the 
most densely populated of the three areas where CII operates and includes the neighborhoods of 
downtown Los Angeles, Rampart, Echo Park, and Silverlake. Its poverty rate is 25 percent, 
higher than average in Los Angeles County. CII also operates the Mid-Wilshire campus in SPA 
4, which is located near Koreatown. As part of CII’s Social Innovation Fund (SIF) grant, CII 
expanded its youth development programs in this service area. 

SPA 6 includes the neighborhoods of Compton, Crenshaw, and Watts in South Central 
Los Angeles, and has the poorest socioeconomic indicators of the three SPAs where CII 
operates. The poverty rate is 31 percent, the highest in Los Angeles County. As part of CIFs 
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Table 2.1 


Key Indicators of Health by Service Planning Area 


Characteristic 

National 

Los Angeles 
County 

SPA 4 

SPA 6 

SPA 8 

Age (%) 

0-5 years 

7.9 

8.0 

7.2 

10.6 

8.1 

6-17 years 

16.2 

16.8 

13.2 

21.0 

16.9 

18-39 years 

29.8 

32.8 

39.1 

34.6 

31.5 

40-64 years 

33.2 

31.6 

30.2 

26.4 

32.7 

65 years or older 

13.0 

10.7 

10.2 

7.4 

10.9 

Race (%) 

Latino 

16.3 

48.1 

52.2 

67.7 

39.5 

White 

64.8 

28.9 

24.9 

2.0 

29.7 

African American 

12.8 

8.5 

4.7 

28.5 

14.8 

Aisan/Pacific Islander 

5.2 

14.3 

18.0 

1.6 

15.8 

American Indian/ Alaskan Native 

0.8 

0.2 

0.2 

0.1 

0.2 

Education (%) 

Less than high school 

14.3 

23.2 

27.6 

38.8 

18.9 

Completed high school 

28.5 

22.3 

22.9 

24.2 

25.6 

Some college, trade school, or associate's degree 

31.2 

27.9 

21.8 

25.5 

28.4 

College or post-graduate degree 

26.0 

26.6 

27.6 

11.5 

27.0 

Poverty (%) 

Household income less than 100% Federal Poverty Level 

15.9 

18.0 

25.0 

31.1 

17.2 
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Table 2.1 (continued) 


Los Angeles 


Characteristic 

National 

County 

SPA 4 

SPA 6 

SPA 8 

Parental support (%) 

Children ages 0-5 years whose parents say they can easily find someone 
to talk to when they need advice about raising their children 

N/A 

87.1 

80.0 

83.9 

89.9 

Child care (%) 

Children ages 0-5 years for whom parents report difficulty finding child 
care (excludes 23.4% who reported they do not need child care) 

N/A 

26.9 

30.4 

36.7 

19.2 


Physical and mental health (%) 

Adults reporting their health to be fair or poor 

Children ages 3-17 years who tried to access mental or behavioral health 
care in the past year 

Adults who tried to access mental health care in the past year 
Adult mental health (%) 


Ever diagnosed with depression 

17.5 

12.2 

13.4 

10.8 

10.7 

Currently diagnosed with depression 

N/A 

8.3 

9.3 

8.0 

7.7 

At risk for major depression 

N/A 

10.4 

11.6 

13.3 

9.3 

Ever diagnosed with anxiety 

N/A 

11.3 

12.0 

10.1 

10.2 

Currently diagnosed with anxiety 

N/A 

6.4 

7.4 

6.9 

5.5 

Insurance (%) 

Children ages 0-17 years who are uninsured 

7.0 

5.0 

6.6 

8.6 

2.9 

Adults ages 18-64 years who are uninsured 

21.3 

28.5 

35.5 

38.2 

26.7 

Regular source of care (%) 

Children ages 0-17 years with no regular source of health care 

3.3 

4.8 

5.2 

7.3 

4.5 

Adults ages 18-64 years with no regular source of health care 

N/A 

23.4 

25.4 

29.4 

21.0 


SOURCE: Los Angeles County Department ofPublic Health (2013). 


16.1 

20.7 

24.5 

30.5 

17.6 

N/A 

7.8 

8.1 

5.8 

8.3 

N/A 

7.5 

8.4 

6.6 

6.5 





SIF grant, CII set out to intensify its efforts and expand and solidify its presence in the Watts 
neighborhood. During the study period, CII staff in SPA 6 operated out of several offices spread 
across the neighborhood and they lacked space for many youth development activities. Howev- 
er, CII has been planning to build a campus similar to the Otis Booth facility in Watts. As a 
relative newcomer to SPA 6, CII has faced challenges in building its presence and earning trust 
in the community. 

SPA 8, located on the southern tip of Los Angeles County, encompasses a much larger 
geographical area than either SPA 4 or 6 and includes the communities of Torrance and Long 
Beach. In 1992, CII expanded its services to SPA 8, where it continues to operate out of 
multiple locations. SPA 8 boasts better socioeconomic indicators than SPAs 4 and 6. Its overall 
poverty rate is just 17 percent. CII’s offices in SPA 8 are more geographically spread out than in 
the other SPAs, and there are no plans to establish a main campus. 

SPAs 4, 5, and 8 are some of the more service-rich areas in Los Angeles County, par- 
ticularly SPA 4. A number of other organizations in these areas have contracts with the Los 
Angeles County Departments of Mental Health and Child and Family Services to serve children 
and families. Similar to CII, these other organizations provide mental health services to children 
and youth. Apart from that, their services vary considerably; they offer different prevention or 
supportive services, mental health services for adults, or services for non-English-speaking 
clients, or they operate at a different scale. Yet while these and other organizations may provide 
similar services throughout Los Angeles County, CII has been described by individuals outside 
the organization as offering a rich set of core mental health services, evidence-based practices, 
and cultural adaptations to better serve clients from Spanish-speaking and several other foreign- 
language communities. 


Overview of Oil’s Services and Operating Philosophy 

CII’s services are roughly divided into four programmatic categories: clinical mental health 
services, family support, youth development, and early childhood care and education. 

• Clinical mental health services include diagnostic treatment and needs as- 
sessment, individual and group therapy, and family therapy. Licensed thera- 
pists or psychologists typically deliver these services. They may include evi- 
dence-based and evidence-informed practices. As discussed later in this 
chapter, most clinical services are provided through contracts with the Los 
Angeles County Departments of Mental Health and Children and Family 
Services. 
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• Family support includes programs offered to parents or guardians. These 
programs address parent education and child development, and family eco- 
nomic success and stability through case management, home visitations, par- 
enting classes, support groups for fathers and grandparents, financial literacy 
workshops, and job-readiness supports. 

• Youth development includes nonclinical programs for young people of dif- 
ferent ages that address life skills, social skills, literacy and education, crea- 
tive arts, and health and wellness. 

• Early childhood care and education services are for infants and children 
from birth to 5 years of age. They include Head Start and child care pro- 
grams. Though early childhood programming encompasses more than one- 
fifth of CII’s overall budget, these services were not the focus of this SIF ini- 
tiative, which targeted youth ages 9 to 24 years. 

As an operating philosophy, CII coordinates the services it provides to meet the holistic 
needs of children and their families. This approach stands in contrast to the fragmented services 
that often characterize the child welfare system. CII conceptualized this philosophy around three 
components that work together to support a child’s well-being: recovery, resiliency, and 
readiness. 


• Recovery from adverse childhood experiences involves reducing the effects 
of trauma and high-risk behaviors. Recovery is the primary focus of CII’s 
clinical services. Programs and services that support parents and guardians in 
helping their children also support recovery. 

• Resiliency is the capacity of children and families to persevere and prevent 
the effects of trauma and is developed by enhancing their protective factors 
and reducing risks. 1 It is the primary focus of early childhood, family sup- 
port, and youth development programming. 

• Readiness for success in school, work, and life involves positive and healthy 
personal behaviors and social relationships, engagement in education or oc- 
cupational training, and the ability to connect to needed supports or re- 
sources. CII’s combined services support readiness. 


'Protective factors are characteristics of individuals, families, or communities that mitigate risks to health 
and well-being. Examples include positive social connections, parenting skills and knowledge of child 
development, and effective communication practices. 
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Figure 2.2 depicts the conceptual framework of these services and their intended short- 
and long-tenn outcomes. Underlying mediating factors affect the successful delivery of these 
services, including the effectiveness of and providers’ fidelity to the evidence-based and 
evidence-informed practices, the trauma lens that the service model applies to all its compo- 
nents, the funding context, and the clients’ characteristics. 

Structure of Clinical Mental Health Services 

Clinical mental health services fall into two main programmatic category, which are as- 
sociated with specific funding streams. Each category includes a variety of therapeutic ap- 
proaches, which may or may not be evidence based: 

• Community Mental Health Services are typically for clients with less- 
intensive needs. Clients see a therapist at a CII office, typically weekly, and 
may receive an evidence-based or evidence-informed practice. The treatment 
may be a standalone service, though some families may receive some case 
management services from a care coordinator at CII. 

• Intensive In-Home Services tend to be for higher-need clients who likely 
have an open case with the Los Angeles County Department of Children and 
Family Services. The Los Angeles County Department of Mental Health or 
Department Children and Family Services supports these services, which 
Box 2.1 describes in more detail. Several different service programs belong 
in this category, including Full Service Partnership, Field Capable Clinical 
Services, and Wraparound services. 2 Though the specific components of 
each program vary, children and their families generally receive therapy, in- 
tensive case management, and support from a team of staff who are available 
24 hours a day, 7 days a week. 

Structure of Family Support 

Parents or guardians of children participating in any mental health service program may 
receive CII’s supportive services or enroll in a CII program or class not directly connected to the 
children’s clinical services. For example, parents or guardians of children enrolled in Full 
Service Partnership, a clinical service, receive supportive services through assigned “parent 
partners,” who coach them on how to better support their children, such as on how to advocate 
for their children at school or better deal with behavioral issues. Examples of programs in which 


2 The Los Angeles County Departments of Mental Health and Children and Family Services also have 
contracts with other organizations in the county that provide these same services. 
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Figure 2.2 

CII’s Integrated Service Model Conceptual Framework 
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Box 2.1 


Programs of the Los Angeles County Departments of Mental Health 
and Children and Family Services that Cll Provides 

Wraparound is an intensive services program designed for high-risk youth in foster care 
placement, at risk of placement, or in urgent need of mental health services. The Los Angeles 
County Department of Children and Family Services or juvenile justice services refer youth to 
the program. The Wraparound team consists of a therapist, a child and family specialist, and a 
parent partner, and meets regularly with the family in home. The team is also on call 24 hours 
a day, seven days week, working collaboratively to ensure that the child lives in a safe and 
permanent home environment. Cll generally provides Wraparound services for 18 months, but 
sometimes for up to two years. 

Full Service Partnerships are designed for underserved children from birth to age 15 years. 
The Los Angeles County Department of Mental Health refers these children, many of whom 
are homeless, have been hospitalized, or are at risk for hospitalization. The Full Service 
Partnership team includes a therapist and a parent partner, who provide crisis intervention 
services and family counseling 24 hours a day, seven days week. Full Service Partnerships are 
designed to do “whatever it takes” to help families secure their well-being, safety, and stability. 
Full Service Partnerships typically last from 9 to 12 months but families can receive services 
for up to two years. 

Family Preservation provides support for families after an allegation of child abuse or neglect 
that is determined to be unsubstantiated. The service focuses on helping families remain 
together, live safely in the same home, or work toward family reunification. 

Field Capable Clinical Services are intensive services for families who are unable to access 
services in a traditional mental health care setting. Services may be provided in the home, 
schools, or community centers. They are appropriate for clients who are lower risk than those 
receiving Wraparound or Full Service Partnership services, but who may have experienced 
trauma and require intensive support. Field Capable Clinical Services can be used to transition 
clients completing a Full Service Partnership by allowing them to continue working toward 
their recovery goals. 

Intensive Field Capable Clinical Services are services offered through a program launched 
in 2013, to accelerate access to Medi-Cal services for children in need of intensive mental 
health services. Children from birth to age 15 years receive immediate access to Intensive 
Care Coordination as well as Intensive Home Based Services, two interventions that address 
serious mental health conditions. Intensive Field Capable Clinical Services are provided within 
24 hours of referral and are available to clients 24 hours a day, seven days a week. 

*Los Angeles County Department of Mental Health (2015). 
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parents or guardians may participate regardless of whether or not their children are receiving 
clinical services include parenting programs such as Project Fatherhood and financial literacy 
classes. 


Structure of Youth Development Services 

Children and youth can access youth development services whether or not they are re- 
ceiving clinical services. Examples of these activities include theater, drawing, or dance classes 
and soccer or basketball. As Chapters 3 and 4 describe in more detail, youth development 
services may precede a client’s enrollment in clinical services or be a client’s “step-down” 
support, which a client receives after completing clinical services. Eligibility requirements for 
youth development services are less restrictive than they are for clinical services, and these 
activities particularly target residents of the neighborhoods near CII locations. 

Combining Services Through the Integrated Service Model 

Through its Integrated Service Model, CII attempts to coordinate clinical, family sup- 
port, and youth development services to best address the complex needs of the children and 
families it serves. Clients may receive multiple types of services, depending on their needs. The 
Integrated Service Model aims not to simply offer multiple types of services but to eliminate 
operating silos among CII’s various services and to create a system that accurately identifies 
clients’ full range of needs and ensures they receive all the support required to address those 
needs. Chapter 4 describes the implementation of the Integrated Service Model in depth. 


CII’s Funding Structure 

How CII funds its services has a major impact on how it implements those services as well as 
the Integrated Service Model. Policymakers at the federal, state, and county levels make 
decisions about what services to provide and how to pay for them, and these decisions trickle 
down to direct service providers such as CII. While the Integrated Service Model aims to 
coordinate the fragmented services typical of the child welfare system, CII must work within 
that system to fund its services, weaving together a complex set of contracts, grants, and other 
sources of revenue. 

Staff at CII and partner agencies described how the funding structure, including state- 
and county-level policies, has a huge impact on what services they are able to provide. For 
example, California’s Mental Health Services Act allocates funds to county mental health 
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departments for Prevention and Early Intervention services, 3 of which Los Angeles County 
dedicates 20 percent of these funds to evidence-based practices. CII is one recipient of these 
funds. Box 2.2 summarizes important laws and policies in California that affect child welfare 
services. 

The majority of CII’s revenue (84 percent) comes from contractual allocations and ser- 
vice fees associated with providing services. These revenue streams include allocations through 
contracts with the Los Angeles County Departments of Mental Health and Child and Family 
Services and reimbursements for services from clients’ health insurers, most often Medi-Cal, 
California’s health coverage for low-income children and adults. 4 The remaining 16 percent 
comes mostly from fundraising efforts. 

CII has numerous contracts to provide services with several agencies, each of which has 
its own requirements. CII’s single largest source of revenue comes from its contract with the 
Los Angeles County Department of Mental Health, which allocates funding through 1 8 differ- 
ent streams or “buckets.” Each of these buckets has requirements about eligible services and the 
target population that receives those services. In addition, each bucket may have further re- 
strictions on how funds can be used. Spending all the allocations from contracts with multiple 
buckets and complex requirements can be a challenging calculus, and CII is always at risk of 
not fully expending these allocations. For example, CII may encounter a situation in which it 
would like to offer a client a particular evidence-based treatment but it does not have enough 
funding in that bucket. Conversely, CII may not be able to use all the funding available in a 
bucket because it does not have enough staff to deliver the full amount of a specific service or 
treatment that the available funding covers. 

Funding constraints can also affect which referrals CII can accept, since CII may re- 
ceive referrals for clients who need services for which it does not have enough funding to 
provide. In these cases, CII will refer the clients to another organization. On the other hand, CII 
can find itself at risk of not spending all available funding eannarked for specific services if it 
lacks clients who meet the stipulated requirements, or if clients drop out of treatment toward the 
end of the contract period. 

Ultimately, funding constraints present huge obstacles to CII in achieving the outcomes 
envisioned by the Integrated Service Model, which requires nonclinical, or “add-on,” supports 
such as youth development activities and programs for parents and guardians. Many of these 


^Prevention and Early Intervention services are targeted at persons who are at the early stages of mental 
illness, before official diagnosis of mental illness or in the very early stages. Services are generally low 
intensity. 

4 The majority of CII’s clients are covered through Medi-Cal. 
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Box 2.2 


California Policy Context 

The Mental Health Services Act (MHSA),* passed in 2004, was designed to expand county 
mental health services across the state of California. The act requires 20 percent of funds to be 
used for Prevention and Early Intervention services, many of which assist children. It also led 
to the implementation of new services, including Full Service Partnerships. In Los Angeles 
County, a portion of MHSA funding is used for evidence-based practices. 

Medi-Cal 1 is the California Medicaid welfare program and provides support to people with 
limited ability to pay for health coverage, including low-income adults, families with children, 
children in foster care, and former foster youth. Medi-Cal beneficiaries are members of their 
county’s Mental Health Plan, through which they can access county-administered mental 
health services. 

California Work Opportunity and Responsibility for Kids (CalWORKs), California’s 
welfare program, provides assistance to families with children who have been deprived of 
parental support. CalWORKs participants have access to Medi-Cal (Medicaid) and other 
programs, through which they may receive mental health services for themselves or their 
children. In 2013, CalWORKs provided assistance to 50 percent of children living in poverty 
in California. 

Child Abuse Prevention, Intervention and Treatment (CAPIT)^ was a law passed in 1982 
aimed at preventing and treating child abuse and neglect. It provides a funding stream for 
programs that provide support services such as child care, mental health services, parental 
education and support, counseling, and screening. Priority for CAPIT funding is given to 
private nonprofit agencies that serve children at risk of abuse or neglect. Counties are respon- 
sible for monitoring projects and reporting annually to the state’s Office of Child Abuse 
Prevention. 

Realignment, 11 a state law enacted in 1991, shifted responsibility for running mental health 
programs from the state to the county level. It was intended to provide a stable funding source 
and client-centered approach to mental health services. In 201 1, a second realignment allocat- 
ed a portion of a state sales tax to each county’s substance abuse and children’s mental health 
programs, making counties responsible for funding and administering their mental health 
programs. 

(continued) 

add-on services are not billable to their clinical funding streams, which is CII’s main source of 
revenue, making grants and other private support critical to the implementation of the Integrated 
Service Model. The SIF grant has been essential to the Integrated Service Model, as it was the 
primary support for CII’s expansion of its youth development activities. 
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Box 2.2 (continued) 

Katie A. v. Bonta, # a lawsuit settled by the state of California in 201 1, dictates how services 
are provided to children in foster care or at risk of entering foster care. The settlement calls for 
California Department of Health Care Services to implement statewide standards and monitor 
mental health services. As a result of the lawsuit, California made available three types of 
mental health services: Intensive Home-Based Services, Intensive Care Coordination, and 
Therapeutic Foster Care. An additional program, Intensive Field Capable Clinical Service, was 
launched to facilitate access to these services. 


Arnquist and Harbage (2013). 

'California Department of Health Care Services (2016). 
^California Department of Social Services (2014). 
^California Department of Social Services (2013). 
Arnquist and Harbage (2013). 

^California Department of Health Care Services (2016). 


History and Development of Evidence-Based Practices at Cll 

CII is widely considered to be a leader in implementing and delivering evidence-based practic- 
es. As an early adopter, CII first began incorporating evidence-based practices into its clinical 
model in 1999, focusing on just one model. Before that, according to one CII staff member, 
“people were just doing general therapy in their offices.” Staff members reported that the major 
shift to evidence-based practices started in 2004 when CII joined the National Child Traumatic 
Stress Network and became involved in its efforts to develop and disseminate evidence-based 
practices for children recovering from trauma. 

Internal and external forces drove CII’s shift to incorporate evidence-based practices in- 
to its clinical model. Internally, senior managers had bought into and championed the approach. 
Some staff members saw evidence-based practices as a way to provide better, more targeted 
services to clients. As one staff member put it, “[ejvidenced-based treatments allowed us to do 
more focused treatment and to ask the right questions early on.” 

Externally, the Los Angeles County Department of Mental Health had been pushing for 
shorter-term, evidence-based treatment models for quite some time, and this pressure intensified 
in 2009, when the Mental Services Act began allocating additional funding for Prevention and 
Early Intervention. “Now, it’s financially essential. You have to be able to do that [evidence- 
based practices] or you’re going to miss out on a tremendous amount of funding,” explained a 
manager at CII. However, staff members also reported that, as an early adapter, CII made the 
transition to evidence-based practices more easily than did other organizations in Los Angeles. 


25 





Even though CII had begun integrating evidence-based practices relatively early and 
senior managers and some therapists championed them, other therapists resisted their wide- 
spread integration. Those who were involved in the early efforts to use evidence-based practices 
described how some therapists preferred the freedom of general therapy or viewed the push for 
evidence-based practices as a criticism of how they were doing things. Others reportedly 
perceived the use of evidence-based practices as a “cookbook” approach and incompatible with 
the personalized relationships they were trying to build with their clients. Staff members also 
recalled some therapists having difficulty implementing the practices, either because they did 
not have the skills or because they found it difficult to focus on a new treatment model while 
dealing with clients’ crisis situations. 

CII managers employed a number of strategies to encourage therapists to adopt evi- 
dence-based practices. They offered therapists incentives early on, such as promotions to those 
who became proficient in evidence-based practices. They also gave therapists a choice of 
practices from which they could select the one that best suited their individual style and ap- 
proach to mental health care. 

Implementing and delivering evidence-based practices still has its challenges. As one 
manager at CII asked, “[h]ow do you get 100 different therapists to do it right and not drift?” 
Additionally, the Los Angeles County Department of Mental Health periodically revises its list 
of approved evidence-based practices, forcing CII to adjust existing treatments it offers, or add 
new ones. Changing or adopting new evidence-based practices, however, requires an invest- 
ment of resources, since each treatment model has specific training and implementation proto- 
cols. Funding for evidence-based practices is also limited. While CII receives funding dedicated 
to evidence-based practices, much of its funding is not. Additionally, evidence-based practices 
are highly specified and may not be appropriate for every client. 


Conclusion 

This chapter presented an overview of CII’s organizational structure, including the services it 
provides and how it funds those services. It has shown that each of CII’s structural components 
are inter-related — the Integrated Service Model dictates how CII organizes staff and services, 
funding affects which services CII provides, and the services CII provides influence what roles 
staff members play. The next chapter describes the population CII serves and how clients enter 
services. 
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Chapter 3 

Pathways to CM 


This chapter presents an overview of the target population that Children Institute, Inc. (CII), 
serves, with an emphasis on 9- to 24-year-olds, and how clients enter services. It describes the 
client referral and assessment process as well as the clients that CII most recently served. CII 
provides services to demographically diverse clients of all ages, at every stage of life, and with a 
wide range of health care needs. 


CII’s Target Population 

Each year, CII serves more than 20,000 children and family members. Many of these clients are 
served through one of CII’s one-day events, such as career fairs, tax preparation services, and 
sporting events, or through child care or Head Start programs. 1 In 2013, CII provided over 
6,500 people with ongoing clinical, family support, or youth development services. 

CII serves clients of all ages. From early childhood programs to Grandma’s House, a 
support group for grandparents raising grandchildren, CII helps children and their families at 
every stage of life. As shown in Table 3.1, more than a third of clients were under 9 years of 
age during the evaluation’s 2012 and 2013 follow-up period, reflecting CII’s expansive early 
childhood programs. Clients ages 9 to 24 years, the population that the Edna McConnell 
Clark Foundation Social Innovation Fund targets, also made up more than a third of CII’s 
clients in the same period, with the majority between 9 and 13 years of age. At 15 years old, 
clients age out of several of CII’s clinical programs, including the Wraparound services and 
Full Service Partnership programs, which explains why the majority of clients between 9 and 
24 year of age are 13 and under. A little over one-fourth of clients in the 2012 and 2013 
follow-up period were 25 years old or older. The demographic information presented in Table 
3.1 does not tell the whole story, however, since CII’s service receipt data did not always 
reflect parents or guardians who received concurrent services. For example, parents or 
guardians receiving Trauma-Focused Cognitive Behavioral Therapy may have participated in 
conjoint sessions with a therapist but the research team could not capture their data as distinct 
and separate clients. Since many parents or guardians receive some services in conjunction 
with their children’s services, the number of people over 25 years of age that CII serves was 
likely understated. 


'This analysis does not include clients who received services only though one-day events. 
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Table 3.1 


Demographic Characteristics of CII Clients 


Characteristic 

All CII 

2012 

SPA 4 

SPA 6 

All CII 

2013 

SPA 4 

SPA 6 

Average age (years) 

15.6 

18.3 

11.6 

17.9 

20.7 

12.1 

Age for all clients 3 (%) 

Under 9 years 

37.3 

31.7 

47.7 

33.6 

28.3 

48.7 

9-17 years 

33.7 

31.4 

36.4 

28.6 

26.1 

33.3 

18-24 years 

4.4 

4.9 

2.4 

5.5 

6.4 

3.0 

25 years or older 

23.0 

31.2 

11.3 

30.0 

38.6 

12.3 

Gender (%) 

Male 

51.1 

52.1 

52.9 

52.2 

52.4 

55.7 

Race/Ethnicity (%) 

Hispanic/Latino 

74.1 

85.8 

61.1 

74.4 

83.3 

63.6 

White 

16.9 

7.5 

36.1 

15.9 

7.7 

33.6 

Black 

4.3 

1.7 

1.7 

3.8 

2.2 

1.0 

Asian 

1.1 

1.2 

0.0 

1.7 

2.5 

0.2 

Other 

3.6 

3.9 

1.1 

4.2 

4.4 

1.5 

Sample size 

5,018 

2,348 

958 

6,637 

4,017 

1,112 


SOURCE: CII management information system. 

NOTE: a Age category percentages may not sum to 100 percent because of missing data. 


Table 3.1 shows that client demographics differ across regions. CII serves a relatively 
younger population in Service Planning Area (SPA) 6, with nearly half of the clients under 9 
years of age, mainly because it provides more Head Start services in the area. In SPA 4, CII 
offers more services to older clients, such as fatherhood and other parenting programs. CII 
serves a primarily Hispanic or Latino population. Nearly three-quarters of CII clients across 
locations are Hispanic or Latino, though the percentage varies by region. Most of the clients in 
SPA 4 are Hispanic or Latino, but that proportion drops to a little less than two-thirds in SPA 6. 
Among the target population of children and youth 9 to 24 years old, the average age was 
between 13 and 14 years old (Table 3.2). The racial and ethnic make-up of this age group 
resembled that of the overall CII population. 
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Table 3.2 


Demographic Characteristics of CII Clients (Ages 9-24 Years) 


Characteristic 

All CII 

2012 

SPA 4 

SPA 6 

All CII 

2013 

SPA 4 

SPA 6 

Average age (years) 

13.7 

13.7 

13.5 

13.5 

13.7 

13.0 

Race/Ethnicity (%) 

Hispanic/Latino 

75.7 

90.8 

58.6 

76.1 

87.0 

58.6 

Black 

16.8 

5.1 

38.6 

16.5 

6.5 

37.7 

White 

4.0 

1.1 

1.7 

3.2 

1.9 

0.8 

Asian 

1.0 

1.3 

0.0 

1.7 

2.4 

0.5 

Other 

2.6 

1.7 

1.1 

2.5 

2.2 

2.4 

Sample size 

1,911 

854 

372 

2,261 

1,305 

403 


SOURCE: CII management information system. 

Although not shown in the tables, all of CII’s clients are low income; many are enrolled 
in Medi-Cal, California’s state-operated insurance program for low-income families. CII 
receives some limited funds to support indigent clients who lack insurance and do not qualify 
for Medi-Cal (sometimes because they are not legal residents) or who have lost it temporarily. 

CII clients have a wide range of needs. Those engaged in only community service activ- 
ities may have few therapeutic needs. Clients receiving clinical services, on the other hand, may 
have needs ranging from intensive and requiring services 24 hours a day, seven days week, to 
less intensive and requiring only weekly sessions with a therapist. Data related to clients’ risk 
factors, such as substance use, trauma history, or household characteristics, were not available. 
However, since the Los Angeles County Department of Children and Family Services referred 
many of the clients receiving CII’s clinical services, it is likely that a substantial proportion of 
them had a history of abuse and come from families with high needs and significant risk factors. 

Though CII’s services address a broad range of client needs, there are some needs that 
CII is not equipped to address. For instance, CII does not serve children with mental health 
issues who also have developmental delays or autism. Other organizations that specifically 
focus on development delays serve these children. 


Intake and Assessment 

As a large organization offering a wide range of services to a diverse population with varying 
needs, CII’s system for receiving and assessing referrals is complex. This section describes the 
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various points of entry into CII services, including new referrals to CII and referrals of current 
clients to other services within CII. Figure 3.1 depicts this process. 

CII’s intake staff are the first point of contact for referrals. In each SPA, CII employs 
intake staff who handle all referrals in the geographic area. Referrals may come from families 
directly, case managers at other organizations or agencies, or even from within CII. Referrals 
may be submitted by phone, e-mail, or fax. As Chapter 2 describes, CII’s funding structure 
requires that CII knit together a diverse set of funding streams, with separate eligibility criteria, 
to meet the individual needs of each client. The intake and assessment process is thus a critical 
component of the Integrated Service Model. 

External Referrals 

The Los Angeles County Departments of Mental Health and Children and Family Ser- 
vices have contracts with CII for referrals. Other common sources of clients include service 
providers in the community (other nonprofits, schools, and so on) or referrals from families 
directly. Intake staff said that referrals from the Department of Children and Family Services 
and from families directly were the most common. Referrals for clinical services may be 
requests for general individual therapy or for a specialized treatment or program. Referrals for a 
specific clinical program are subject to the eligibility requirements of that program, which the 
funder often dictates. For example, if a family submits a referral to participate in the Full 
Service Partnership program and does not already have an open Full Service Partnership case 
with the Department of Mental Health, CII intake staff would have to first refer the family to the 
department for approval. As a less intensive program with fewer funding constraints, CII’s 
Community Mental Health services have more flexible requirements for referrals. Unlike 
intensive services, referrals for Community Mental Health services require only CII’s approval 
and not that of the relevant state agency. Though intake staff are ultimately responsible for 
tracking all referrals, specific programs may receive some referrals first. For example, the 
Wraparound services or Full Service Partnership team in a particular SPA may receive a referral 
directly from a caseworker at the Department of Children and Family Services. In this case, the 
program team informs the intake staff, who in turn log and track the referral. 

CII staff members reported schools to be an important source of referrals. One such 
source is CII’s Cognitive Behavioral Intervention for Trauma in Schools (CBITS) program, 
which CII operates only in schools with which it has a relationship. A form of group therapy, 
CBITS only works when there is a critical mass of participants. As such, CII relies on schools to 
identify program participants and obtain parent or guardian consent. A successful referral 
network for CBITS involves training staff at the school on how to identify children who are 
appropriate for the treatment. In some schools, staff may screen whole classes of students to 
identify those for whom CBITS is appropriate. 
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Figure 3.1 

Basic Client Flow Through CII Services 
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According to CII staff members, referrals to youth development and family support ser- 
vices come from a wider array of sources than do clinical service referrals and involve more 
residents of the neighborhoods immediately surrounding CII locations. Sources of these 
referrals include local schools, neighborhood outreach such as informational handouts, and 
word of mouth. In SPA 4 where CII operates its flagship facility, staff reported that area 
residents often walked in off the street to inquire about services. “They see our wonderful 
building,” explained one staff member. “They see people playing basketball outside. They come 
off the street, and they come and find our service because we’re the most low barrier. We’re 
easy access. We’re in their neighborhood.” Intake staff may also refer individuals or families 
who initially requested clinical services to youth development services if CII cannot address 
their clinical needs immediately or at all. If they request services that CII cannot provide, intake 
staff will refer them to another organization that can. Similarly, if there is no funding available 
or no funding stream for the type of service request received, intake staff will refer the individu- 
al or family to another organization. 

Internal Referrals 

In line with the Integrated Service Model, staff members seek to coordinate CII’s ser- 
vices to respond to clients’ full-range of needs. In doing so, they may refer clients already 
receiving services to additional services within the agency. An individual therapist or member 
of the program team may refer a client receiving clinical services to youth development or 
family support services. In this case, the intake staff conduct an internal referral, whereby they 
contact the client’s family and connect the client to the suggested service. The most common 
internal referrals were those for clients receiving clinical services to youth development 
services. 
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Internal referrals for clients engaged in youth development activities to clinical services 
were less common. Staff explained that because the population participating in youth develop- 
ment services is not typically high need, these types of referrals were less frequent. While a 
contractor often provides youth development services, a CII staff member typically facilitates 
the activities and may identify participants who would benefit from clinical services. In this 
case, the staff member contacts the intake staff, who in turn reach out to the client’s family. 

Case Disposition and Assessment 

CII intake staff in each service area reported that though they may receive dozens of re- 
ferrals each week, they do not process them in the order in which they receive them. Instead, 
they prioritize them based on need. They first assign high-risk cases to a therapist, such as those 
in which clients express suicidal ideations or are cutting themselves. Intake staff also consider 
funding sources when assessing referrals. One intake team described referrals from the Los 
Angeles County Departments of Mental Health or Children and Family Services as “gold card” 
referrals and flagged them for immediate assistance. 

Case disposition teams meet weekly to discuss where they should assign new referrals. 
The team assigns referrals for specific programs, such as Wraparound services and the Full 
Service Partnership, to those programs. For referrals to Community Mental Health services, the 
team will assign clients to specific therapists based on the infomiation in the referral. Once 
assigned, the therapist contacts the client and conducts an assessment of the client’s needs. 

Clients may meet their therapists and undergo the assessment at the CII office or in their 
schools or homes. The assessment typically includes the Child/Adolescent Initial Assessment, a 
nine-page tool used by the Los Angeles County Department of Mental Health. This tool gathers 
the client’s medical and psychiatric history and information on the client’s development and 
environmental stressors, academic history, involvement with the juvenile justice system, and 
family structure. 2 Therapists use it to make an initial diagnosis and develop the client’s treat- 
ment plan. 3 Therapists also use the Global Assessment of Functioning scale to rate the severity 
of the client’s mental illness. Clients must receive a score of 50 or below (on a scale of 0 to 100) 
to be eligible for services covered by Medi-Cal. 

Clients may undergo additional assessments depending on what program they enter or 
treatment they receive and its respective funding source. For example, Wraparound services 
team uses an extensive screening tool that breaks down a client’s needs by domains, such as 


2 Los Angeles County Department of Mental Health (2013). 

’During site visits, the research team learned that CII was planning on implementing a new assessment 
tool called the Universal Screening Tool. The report does not discuss it because CII had not fully implemented 
it during the evaluation period. 
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family, educational, and emotional needs. If this initial assessment determines that a client is 
appropriate for an evidence-based practice, the client may undergo additional assessments 
associated with that treatment. Functional Family Therapy (FFT), for example, has its own 
assessments that clients must undergo during the first phase of treatment. For clients receiving 
Community Mental Health services, a care coordinator will typically join the first assessment 
meeting with the therapist to leam if there is a need for nontherapeutic supports, such as housing 
assistance. Some of the assessments serve as a baseline measure of the clients’ symptoms and 
are administered again near the end of treatment to assess the client’s progress. Such assess- 
ments include the Youth Outcome Questionnaire and the Post-Traumatic Stress Disorder 
Reaction Index, both of which have components for the child and parent or guardian. 

After the assessment process, the client begins treatment. Depending on the treatment 
plan, the client may continue treatment with the same therapist who administered the assess- 
ment, or may be reassigned. For example, if the client discloses a traumatic experience during 
the assessment process that had not been disclosed on the referral, the client may be reas- 
signed to a therapist with an expertise in that particular area. The client’s family may also 
decide after the initial assessment that it wants to change the treatment plan. For example, 
after starting FFT, a family may decide that a family approach is not appropriate and request a 
different type of service. 

CII often has waitlists for some of its clinical services, particularly Community Mental 
Health services, which receive more referrals for individual therapy from families directly. Staff 
noted that the more established CII offices in SPA 4 tend to have waitlists because people in the 
community know them and seek them out for services. SPA 6 staff reported that they often have 
a waitlist for the Full Service Partnership program because CII is among only a few organiza- 
tions in the area who serve clients from birth to age 15 years. 

One strategy that CII staff use to shorten or eliminate waitlists is to refer waitlisted cli- 
ents to other types of services. For example, staff members may refer clients waitlisted for 
clinical services to family support or youth development services. Such referrals may not 
always be appropriate, however, since clients with severe mental health needs may not have the 
social skills to engage in these activities. 

Changes to the Intake and Assessment Process 

Over the course of the research team’s three site visits in 2013, CII was in the process 
of changing its client intake and assessment procedures to better align them with the Integrat- 
ed Service Model. During the first two site visits, staff identified a number of challenges with 
these procedures. One challenge, described earlier, was that decisions made by the case 
disposition team were not always appropriate for the clients. Another challenge was that 
internal referrals sometimes resulted in a client falling through the cracks, as staff “passed the 
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baton” and did not follow up to ensure the client received services. A third challenge was that 
intake staff lacked the qualifications to handle cases involving an immediate need, such as a 
client who requires hospitalization. 

To address some of these challenges, CII developed and began implementing the Fami- 
ly Engagement Team model during the evaluation period. However, the research team could not 
fully assess its implementation in this evaluation since it was fully operational only in SPA 4 
and only during the final site visit. The Family Engagement Team model introduced important 
changes to the intake and assessment process. First, CII hired a full-time supervisor to oversee 
the intake staff and who had the qualifications to refer clients with immediate needs such as 
those requiring hospitalization. Second, the model established regional case disposition meet- 
ings. Previously, individual programs, such as Community Mental Health services, held their 
own case disposition meetings, which often stymied coordination. Regional case disposition 
meetings expanded the number of programs represented at these meetings to better integrate the 
delivery of services. A perceived benefit of the Family Engagement Team model is that it helps 
shorten waitlists. With all available programs in a region on the table, CII staff could refer more 
clients on waitlists to other programs and services. At the time of the third site visit, staff 
reported that the introduction of the Family Engagement Team had reduced the waitlist by up to 
60 percent. 


Conclusion 

CII serves clients at every stage of life and with a wide range of needs. CII tailors its services to 
the specific needs of each client and client’s family, determining the most appropriate and 
effective treatments and coordinating them with other services and activities. CII’s complex 
funding structure makes the client intake and assessment process a critical component of its 
Integrated Service Model. The next chapter describes in detail CII’s services and the strategies it 
uses to meet clients’ needs. 
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Chapter 4 

Implementation of CM Services 


The previous chapters describe the context in which the Children’s Institute, Inc. (CII), operates, 
its organizational characteristics, and the client intake and assessment process. In this chapter, 
the focus turns to the implementation of CII’s multiple service categories: clinical mental health 
services, particularly evidence-based practices; family support services; and youth development 
services. 1 Vignettes of CII client experiences compiled from the discussions with their thera- 
pists, information from their case files, and analysis of data from CII’s management information 
systems are included throughout the chapter to present a clearer picture of CII’s services. 2 

As described earlier, CII serves a high-need population, which would otherwise typical- 
ly receive fragmented services through the child welfare system. CII offers its clients a wide 
range of services — both clinical mental health services and nonclinical activities — to address 
their many needs. As presented in Table 4.1, over 40 percent of the more than 6,500 clients 
served during the study period were engaged in clinical mental health services; 3 of these clients, 
about one-third were engaged in an evidence-based treatment and over 20 percent were engaged 
in an evidence-informed practice. 4 Furthermore, over 50 percent of all clients received family 
support services while over 60 percent participated in youth development activities. Each of 
these services is described in more detail below. 

While CII offers clients a variety of services and activities, its operating philosophy is to 
coordinate these services to meet the holistic needs of the child and family. The philosophy 


1 A fourth service category, which encompasses a substantial portion of CII’s resources, is early childhood 
care and education; however, it is not a focus of this report. 

‘In 2014, CII replaced the management information system from which the data for this report was pulled. 
The limitations of the old system prevented certain types of analyses, which this report describes. There were 
also some data quality issues, which limited the types of analyses that the research team could perform. CII 
expects the new system, which incorporates new checks and balances to improve data quality, to alleviate 
many of these challenges. 

’Analysis is limited to any client who enrolled from January 1, 2012, through June 30, 2013, and received 
services from January 1, 2012, through December 31, 2013. 

4 These categories are not mutually exclusive. Evidence-based practices include the following treatments 
and programs: Cognitive Behavioral Intervention for Trauma in Schools, Child-Parent Psychotherapy, 
Functional Family Therapy, Incredible Years, Managing and Adapting Practice, Multidimensional Treatment 
Foster Care, Parent-Child Interactive Therapy, Reflective Parenting Program, Trauma-Focused Cognitive 
Behavioral Therapy, and Trauma Systems Therapy-Substance Abuse. Evidence-informed practices include the 
following treatments and programs: Domestic Violence Treatment Groups, Project Fatherhood, Wraparound 
services. Youth with Sexual Behavior Problems, and social skills and parent support groups. Some level of 
evidence informs these practices but not as much as evidence-based practices have accumulated. 


35 



Table 4.1 


Client Participation in Any CII Services 


Characteristic All CII 

Participation in any clinical services 3 (%) 44.1 

Ever engaged in evidence-based practice 13 32.8 

Ever engaged in evidence-informed practice 0 23.7 

Participated in any family support services (%) 54.8 

Participated in any youth development services (%) 61.1 

Sample size 6,566 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, 
through June 30, 2013, and received services from January 1, 2012, through 
December 31, 2013. 

This analysis excludes one -day youth development events. 

These characteristics are not mutually exclusive. 

a These measures represent the percentage of clients among all those who 
belong to this subset of CII clients. 

b Evidence-based practices include the following treatments and programs: 
Cognitive Behavioral Intervention for Trauma in Schools, Child-Parent 
Psychotherapy, Functional Family Therapy, Incredible Years, Managing and 
Adapting Practice, Multidimensional Treatment Foster Care, Parent-Child 
Interactive Therapy, Reflective Parenting Program, Trauma-Focused Cognitive 
Behavioral Therapy, and Trauma Systems Therapy for Substance Abuse. 

c Evidence-informed practices include the following treatments and programs: 
Domestic Violence Treatment Groups, Project Fatherhood, SOAR, Wraparound 
services, and Youth with Sexual Behavior Problems. 


consists of three core principles that inform all of CII’s operations: recovery, resiliency, and 
readiness. 

This chapter describes CII’s staffing structure and the three categories of service it pro- 
vides. It then explains, how through the Integrated Service Model, CII coordinates these 
services to create a system of care for each client and client’s family. 
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Staffing Structure 

CII operations in each Service Planning Area (SPA) have a similar staffing structure. Figure 4.1 
presents an organizational staffing chart. 5 A regional vice president oversees all services and 
activities in an SPA. To prevent isolation and encourage information sharing among SPA 
operations, CII convenes regular cross-area management meetings of the vice presidents. 
Regional directors report to the vice president. Each SPA operation has multiple regional 
directors, who are responsible for different aspects of service delivery. In each SPA, two 
regional directors oversee clinical services (one responsible for community mental health 
services and the other for intensive in-home services) and one regional director oversees 
community services. The regional directors meet regularly to update one another on develop- 
ments in the different service areas. Therapists, care coordinators, case managers, parent 
partners, and other staff members report to the regional director overseeing the services they 
provide. 


Leaders of each SPA operation report to the CII executive team, which is responsible 
for all CII activities. The team includes the president and chief executive officer, executive vice 
president, and the senior vice president of programs who provides leadership to all of CII’s 
direct service programs. Other executive staff include those overseeing development activities, 
human resources, and facilities management. 


Clinical Mental Health Services 

Licensed therapists (or supervised clinical interns) typically provide CII’s clinical mental health 
services in individual, family, or group settings. 6 In individual therapy, therapists meet one-on- 
one with the client; in family therapy, therapists typically meet with all immediate family 
members together; and in group therapy, one or more therapists facilitates a small group of 
clients. Clinical services range in intensity in any of these settings, and may be delivered as a 
standalone treatment or as part of a multiservice program, such as Wraparound services (an 
intensive, in-home program), depending on eligibility and funding. 

All of CII’s clinical services address the effects of traumatic experiences in the lives of 
children and youth. CII, however, does not take a one-size-fits-all approach to treatment; rather, 
intake and clinical staff assess clients individually to determine the best treatment option for 
their needs. CII also allows for flexibility in treatment in order to accommodate unforeseen 


5 Early childhood care and education has a different staffing structure and is not presented here. 

6 CII is an American Psychological Association-accredited internship site and hosts interns pursing ad- 
vanced degrees in clinical psychology at the University of Southern California School of Social Work. 
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Figure 4.1 

Sample SPA Staffing Organizational Chart 



circumstances. Varying degrees of evidence support CII’s therapies and treatment practices, 
with the strongest research backing its evidence-based practices. Box 4.1 describes several 
particularly prevalent evidence-based practices that the study’s target age group receives. 7 CII 
also uses a number of “evidence-informed” practices that are infonned by some evidence but 
not as much as evidence-based practices have accumulated. Box 4.2 presents examples of these 
practices. Depending on a client’s situation, CII may use usual care practices to treat clinical 
mental health issues since not every issue requires an evidence-based or evidence-informed 
practice. 


7 Los Angeles County contracts with other organizations to provide the same set of services throughout the 
county. Individuals can only seek treatment from one provider at a time. 
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Box 4.1 


Evidence-Based Practices Used by CM 
for the Study’s Target Age Group* 

Trauma-Focused Cognitive Behavioral Therapy (TF-CBT ) 1 is a conjoint child and parent 
psychotherapy approach for children and adolescents ages 3 to 1 8 years experiencing signifi- 
cant emotional and behavioral difficulties related to traumatic life events. 

Functional Family Therapy (FIT) is a family intervention designed for at-risk youth ages 
11 to 18 years experiencing family conflict and behavioral problems. FFT treats the entire 
family as the client and is a short-term treatment, lasting 12 to 14 sessions. 

Incredible Years (IY) is a group parenting program designed to strengthen parent-child 
interactions by teaching parents to provide positive discipline, increase confidence, and 
become involved in their child’s school experiences and social development. 

Cognitive Behavioral Intervention for Trauma in Schools ((BITS) is a school-based 
group intervention for youth ages 10 to 14 years who have been exposed to community and 
domestic violence. Over 10 sessions, students decrease symptoms of post-traumatic stress 
disorder and depression, while building resilience and increasing peer support. 

Multidimensional Treatment Foster Care (MTFC ) 11 is a comprehensive intervention for 
foster children and adolescents ages 7 to 17 years with behavioral problems. Children are 
placed with foster parents who have been trained as part of the treatment team. MTFC seeks to 
help children develop positive relationships and behaviors. 

Reflective Parenting Program (RPP) # is a 12-week workshop series providing parenting 
skill s and instruction to groups of 6 to 10 parents with infants or adolescents. Parents acquire 
skill s and strategies to support their children’s ability to develop and maintain positive rela- 
tionships in the future. 

Trauma Systems Therapy: Substance Abuse (TST-SA ) 5 is an application of the Trauma 
Systems Therapy model with a focus on treating adolescent traumatic stress and substance 
abuse for children or adolescents with ongoing traumatic stressors. The model identifies 
interventions that will improve self-regulation of emotional responses. 

(continued) 
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Box 4.1 (continued) 

Managing and Adapting Practice (MAP)* is a system developed to coordinate and supple- 
ment the use of evidence-based practices to address children’s mental health needs. MAP 
provides support to therapists in treatment decisions, such as selection and delivery of evi- 
dence-based practices. The model measures clients’ progress and assists in adjusting the 
service plan as needed. 

Cll uses other evidence-based practices for younger age groups. These practices include Parent-Child 
Interaction Therapy for children 2 to 7 years old and Child Parent Psychotherapy for children up to 6 years 
old. 

'National Child Traumatic Stress Network (2007). 

^Functional Family Therapy, LLC (2015). 

^Children’s Institute, Inc. (2015). 

MTFC is now known as Treatment Foster Care Oregon. 

#Center for Reflective Communities (2016). 


Box 4.2 

Evidence-Informed Practices Used by Cll 
for the Study’s Target Age Group* 

SOAR, a 10-week social skill development group developed at Cll. During weekly sessions, 
children ages 9 to 12 years, in gender-specific groups of 10 to 12, explore themes including 
individuality, cultural diversity, healthy relationships, teamwork, and leadership. The groups 
are co-facilitated by two adults who guide children through role play and reflection exercises. 
Parents of participating children have access to a concurrent cycle of weekly support meetings. 

Project Fatherhood, a program providing support to fathers of at-risk children who are often 
in the child welfare system. Services offered through the program include therapy, job- 
readiness training, and parenting education. Project Fatherhood aims to increase father in- 
volvement in children’s development and strengthen the father-child relationship. 

Youth with Sexual Behavior Problems, a group for children ages 10 to 14 years who have 
exhibited inappropriate sexual behavior with other children, some of whom have been victims 
of sexual abuse themselves. Over 25 weeks, children learn about the consequences of their 
actions and how to exercise self-control. Parents participate in parallel groups that provide 
guidance in effective strategies for communicating with children about sexuality and safety. 

Domestic Violence Treatment Groups, a 30-week group for adult survivors of domestic 
violence with parallel groups for their children. The groups teach coping and emotional 
regulation skills while helping parents make decisions to protect themselves and their children. 

Children’s Institute, Inc. (2015). 


40 






Table 4.2 presents infonnation about tenure of CII’s clinical services staff and their 
training in evidence-based practices. Only staff members with specific qualifications were 
eligible for training in evidence-based practices, namely those holding a relevant master’s 
degree. 8 The majority of the therapists and psychologists included in this analysis were trained 
in an evidence-based practice, with the percentage increasing between 2012 and 2013. Over 40 
percent of all clinical staff were trained in Trauma-Focused Cognitive Behavioral Therapy (TF- 
CBT), with a much smaller percentage trained in Functional Family Therapy (FFT). 9 The 
average tenure of clinical staff at CII was nearly four years in 20 1 2 (and a bit less in 20 1 3). 

The majority of clients engaged in any of CIFs clinical services are female and Hispan- 
ic/Latino (Table 4.3). Nearly 40 percent of them are 9 to 24 years old — the study’s target age 
group. 10 Table 4.4 presents analysis of the duration and intensity of clients’ participation in 
clinical services (individual or group therapy); this analysis includes clients receiving usual care, 
evidence-based, or evidence-informed practices. Data show that younger clients 9 to 17 years 
old typically participated in individual therapy for nearly seven months, while those 18 to 24 
years old participated for about half of that time. The younger clients also tended to have more 
contact with their therapists than did the older ones. These differences are not unexpected since 
treatment intensity and duration varies based on the specific therapy, and different therapies 
target different age groups. 

Clients participating in group therapy had slightly different experiences depending on 
their age and SPA. Clients 9 to 1 7 years old on average participated in group therapy for about 
the same duration and number of sessions as did clients 1 8 to 24 years old, though there was 
some variation by SPA. While it appears that older clients spent on average less time in each 
group therapy session, this difference is actually the result of how the management infonnation 
system recorded the data, which calculated this time using an equation that factored the number 
of clients in a group as well as the number of therapists facilitating it. Groups for older clients 
tended to be larger, and as a result the time spent in sessions was divided across more partici- 
pants. The average service times for group therapy shown in Table 4.4 thus do not accurately 
reflect the face-to-face time each client spent with a group therapist. 


8 As the notes in Table 4.2 explain, this analysis was limited to therapists or psychologists during the analy- 
sis period and to those for whom training dates were available. 

9 Two-thirds of staff trained in any evidence-based practice were trained in TF-CBT, while 14 percent were 
trained in FFT. 

10 The research team did not expect the large number of clients clustered in the age category of 25 years or 
older. CII does offer a program for adults through California Work Opportunity and Responsibility to Kids 
(CalWORKs), the state’s welfare -to-work program, which accounted for many of these clients. Data entry 
error may also have contributed to this high number. 
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Table 4.2 


Clinical Services Staff Tenure and Training in Evidence-Based Practices 


Characteristic 

All CII 

2012 

SPA 4 

SPA 6 

All CII 

2013 

SPA 4 

SPA 6 

Staff trained (%) 







Evidence-based practices 3 

64.8 

62.8 

63.3 

66.9 

62.9 

67.4 

Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) 

42.8 

42.3 

40.8 

46.4 

41.4 

45.7 

Functional Family Therapy (FFT) 

8.8 

6.4 

14.3 

7.2 

4.3 

13.0 

Average time at CII (months) 

44.8 

51.9 

41.7 

40.7 

47.0 

35.9 

For staff trained in TF-CBT 

45.0 

48.2 

45.4 

40.7 

48.0 

32.7 

For staff trained in FFT 

44.1 

43.8 

34.9 

47.8 

42.7 

38.2 

Sample size 

159 

78 

49 

166 

70 

46 


SOURCE: CII management information system. 

NOTES: This analysis includes only clinical staff who were identified as therapists or psychologists during the study 
period; there are many other staff at CII who support clients in other ways but they are not represented here. Only staff for 
whom training dates were available were counted in this analysis, which likely underreports the percentage of staff who 
are trained. 

These characteristics are not mutually exclusive. 

“This analysis includes staff trained in the following evidence -based practices: Cognitive Behavioral Intervention for 
Trauma in Schools, Child Parent Psychotherapy, Functional Family Therapy, Incredible Years, Managing and Adapting 
Practice, Parent Child Interactive Therapy, Reflective Parenting Program, Trauma-Focused Cognitive Behavioral 
Therapy, and Seeking Safety. 



Table 4.3 


Demographic Characteristics of CII Clients Engaged in Clinical Services 


Characteristic 

All CII 

SPA 4 

SPA 6 

Age a (%) 

Under 9 years 

39.5 

35.3 

48.2 

9-17 years 

33.9 

36.5 

34.7 

18-24 years 

4.5 

5.1 

1.9 

25 years or older 

17.5 

20.6 

9.9 

Gender (%) 

Male 

44.4 

44.7 

47.6 

Race/Ethnicity (%) 

Hispanic/Latino 

71.0 

86.9 

62.6 

Black 

19.0 

6.5 

33.9 

White 

6.2 

2.5 

2.5 

Asian 

0.9 

1.1 

0.0 

Other 

2.9 

2.9 

1.0 

Sample size 

2,778 

1,005 

644 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, through June 30, 
2013, and received services from January 1, 2012, through December 31, 2013. 
a Age category percentages may not sum to 100 percent because of missing data. 


Clinical Mental Health Services Infrastructure 

CII’s clinical mental health services require significant infrastructure dedicated to 
training therapists and supporting their work. All therapists are licensed clinicians (or super- 
vised clinical interns) and have earned at least their masters in marriage and family therapy or 
a related field; some are licensed psychologists or hold advanced degrees in clinical psychol- 
ogy. Newly hired clinical staff must successfully complete extensive training in CII policies 
and procedures and many receive training in specific evidence-based practices. As noted 
earlier, nearly two-thirds of CII staff in 2012 had been trained in an evidence-based practice 
during their tenure. Over 40 percent of staff received training in TF-CBT and less than 10 
percent in FFT. 

In addition to formal trainings, CII dedicates core resources to varying levels of support 
for therapists. Therapists implementing and delivering evidence-based practices typically report 
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Table 4.4 


Duration and Intensity of Client Participation in CII Clinical Services 



Individual Therapy 

Group Therapy 

Characteristic 

All CII 

SPA 4 

SPA 6 

All CII 

SPA 4 

SPA 6 

Number of successive months of Darticiuation 

Age 

9-17 years 

6.8 

6.8 

6.8 

3.9 

3.8 

4.2 

18-24 years 

3.2 

2.6 

5.4 

4.0 

5.3 

2.0 

Averaee number of contacts Der participant 

Age 

9-17 years 

21.3 

20.0 

22.7 

10.9 

12.1 

9.3 

18-24 years 

9.3 

5.2 

18.5 

9.9 

12.3 

4.0 

Averaee service time per contact a ’ b " c (minutes) 

Age 

9-17 years 

99.6 

92.4 

102.2 

35.9 

30.4 

64.0 

18-24 years 

77.6 

82.1 

83.7 

28.8 

27.2 

135.0 

Averaee service time per participant 0 (hours) 

Age 

9-17 years 

35.4 

30.8 

38.7 

6.5 

6.1 

9.9 

18-24 years 

12.1 

7.1 

25.9 

4.8 

5.6 

9.0 

Sample size 

1,054 

423 

249 

297 

127 

16 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, through June 30, 2013, and 
received services from January 1, 2012, through December 31, 2013. 

Youth engaged in family -based therapy were counted as participating in individual therapy. An 
example of group therapy is Cognitive Behavioral Intervention for Trauma in Schools. 

“Average service time per contact (minutes) was calculated using a different unit of analysis from the 
unit of analysis used to calculate other measures in the table. For this measure, data are presented as an 
average per contact, while other measures are presented as averages per participant. 

b Service time per contact for group therapy was calculated by dividing group session time by the 
number of participants in the group, for billing purposes. As a result, reported service times and 
calculated averages are lower than the actual length of the group session. 

c For sessions that require a therapist to travel to a client's school or home, service time includes travel 
time; session time includes documentation time for all clients. 
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to multiple supervisors and attend multiple supervision meetings each week. Therapists meet 
with their clinical supervisors monthly. (Interns meet with theirs weekly until they are 
licensed.) During these meetings, they review their caseloads and discuss any challenges they 
may be experiencing, such as difficulties working with a particular family or problems 
properly documenting services in CII’s management information system. The clinical super- 
visor is also responsible for evaluating therapists’ and interns’ performance, recommending 
them for promotions, and dealing with any discipline issues. Therapists also attend supervi- 
sion meetings specific to evidence-based practices they use. For example, therapists providing 
TF-CBT attend either a weekly or bi-weekly supervision meeting depending on their level of 
expertise in the therapy. 

Despite this support system, therapists in different programs reported some shared chal- 
lenges to delivering clinical services. Los Angeles County Departments of Mental Health and 
Children and Family Services require that therapists complete a significant amount of documen- 
tation for each client. Nearly 80 percent of the staff who responded to the research team’s 
survey noted that it was somewhat to extremely challenging to complete documentation in a 
timely manner. In interviews with the research team, staff described using multiple evidence- 
based practices as particularly challenging, especially if the practices differed theoretically. 
They explained that each evidence-based practice requires additional documentation and its 
own supervision meetings, making the delivery of more than one burdensome. Regardless of 
the practices used, staff across programs described how treating young people with complex 
and multiple traumas and who often lived in perpetual crisis made it difficult to adhere strictly 
to a treatment plan. 


Family Support 

Family support services are those offered to parents and guardians to improve their capacity 
to support their children. CII offers many of these services to members of the larger commu- 
nity regardless of whether or not they participate in CII’s clinical services. Designed to help 
families achieve economic stability, develop positive parenting practices, and connect with 
other families, these services are frequently offered in English and Spanish and include 
support groups for grandparents raising grandchildren, fatherhood classes and other parenting 
classes, financial literacy workshops, and income tax preparation. Clinicians reported in the 
survey that they referred parents or guardians to support groups and special family events 
among other activities. 

CII offers some family support services in conjunction with clinical services. For ex- 
ample, families participating in Wraparound services or the Full Service Partnership program 
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receive support services through their assigned parent partners, who as part of these programs 
assist individual parents or guardians with a range of issues. 1 11 Clients receiving clinical services 
may also access case management services. Some evidence-based practices such as TF-CBT 
require clients and their parents or guardians to attend conjoint or collateral therapy sessions 
together. And while CII offers FFT as a clinical service, it is generally classified as a family 
support service as well. 

A variety of staff deliver CIFs family support services. Therapists provide family-based 
therapy such as FFT and work directly with their clients’ parents or guardians to help them 
support their children’s recovery. Care coordinators, case managers, and parent partners also 
provide family support services. They often work together as part of a care team in Wraparound 
services or the Full Service Partnership program and help parents or guardians advocate for 
their children at school, secure basic needs, or address their children’s behavioral issues. 

The research team encountered several challenges when analyzing the available data re- 
lated to CIFs family support services. As mentioned earlier, CII offers some family support 
services as part of clinical services, which made it difficult to distinguish them in the manage- 
ment information system data. Similarly, records of family support services were associated at 
times with the client and at other times with the parent or guardian in an unsystematic way, 
again making it difficult to determine the precise service dosages. For example, Table 4.5 
indicates that one-third of clients engaged in family support services were over 25 years old; 
however, most of these clients were parents or guardians participating in CII’s Project Father- 
hood or another parent education group. 


Youth Development 

Youth development services are those offered to children and youth to build resiliency, enhance 
protective factors, and reduce risks and perhaps the need for mental health care in the future. As 
one CII staff member said, “[i]f you do not also instill in kids the ability to develop, build self- 
esteem, and imagine a better future for them, their success will be limited.” Similar to family 
support services, these activities target youth and families living in neighborhoods immediately 
surrounding CII locations. CII’s youth development activities focus on life and social skills, 
literacy and education, creative arts, and health and wellness and help young people develop 
self-esteem and social skills. 


1 'Parent partners provide emotional support to parents and serve as links to community resources, among 

other things. One parent partner described working with a parent to address a child’s poor hygiene; another 
described working with a parent to get a child to do household chores. 
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Table 4.5 


Client Demographic Characteristics and Participation in Family Support Services 


Characteristic 

All CII 

SPA 4 

SPA 6 

Participates in any family support 

Age (%) 

Under 9 years 

30.7 

20.0 

45.6 

9-17 years 

25.6 

20.9 

32.7 

18-24 years 

6.5 

8.9 

2.8 

25 years or older 

33.7 

48.8 

13.9 

Gender (%) 

Male 

57.2 

62.1 

55.1 

Race/Ethnicity (%) 

Hispanic/Latino 

69.8 

78.1 

62.3 

White 

17.8 

9.2 

34.3 

Black 

5.3 

2.7 

2.2 

Asian 

1.1 

1.1 

0.0 

Other 

6.0 

8.8 

1.2 

Sample size 

3,468 

1,776 

669 


(continued) 


As part of a Social Innovation Fund (SIF) grant it received, CII sought to expand its 
youth development services during the study period. This expansion of services was particularly 
evident in SPA 4. 12 While CII in other SPAs offered some youth development activities, those 
provided in SPA 4 were far more comprehensive. 13 In addition to expanding services, youth 
development staff also modified their approach to service delivery, in part by starting all group 
activities at the same time and running them for the same duration. 14 CII offers most activities to 
specific age groups, from elementary schoolers through teenagers, typically at 10- week inter- 
vals. These activities include sports such as soccer and basketball and classes in theater, dance, 


12 Most of the expansion of services in SPA 6 occurred after the study period and thus is not reflected in 
this report. 

13 CII began developing a flagship facility in SPA 6 during the study period but the data do not account for 
this expansion. 

14 CII also attempted to implement pre- and post-tests to measure changes in outcomes for community 
services, but encountered some challenges. 
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Table 4.5 (continued) 


Characteristic 

All Cll 

SPA 4 

SPA 6 

Number of successive months of participation 

Age 

9-17 years 

6.9 

6.7 

6.7 

18-24 years 

4.1 

3.8 

6.5 

Average number of contacts per participant 

Age 

9-17 years 

20.7 

18.8 

22.3 

18-24 years 

8.1 

7.0 

12.9 

Average service time per contact a " b (minutes) 

Age 

9-17 years 

67.4 

52.2 

78.5 

18-24 years 

16.9 

12.9 

15.6 

Average service time per participant ' 1 (hours) 

Age 

9-17 years 

23.2 

16.3 

29.2 

18-24 years 

2.3 

1.5 

3.3 

Sample size 

1,153 

537 

250 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, through June 30, 2013, and 
received services from January 1, 2012, through December 31, 2013. 

“Average service time per contact (minutes) was calculated using a different unit of analysis from 
the unit of analysis used to calculate other measures in the table. For this measure, data are presented as 
an average per contact, while other measures are presented as averages per participant. 

b For sessions that require a therapist to travel to a client's school or home, service time includes 
travel time; service time includes documentation time for all clients. 


cooking, and art. Box 4.3 presents more examples. CII also holds a variety of one-day events for 
young people and their families, such as field trips, movie nights, holiday celebrations, and 
college or career fairs; this analysis does not include these events. 

Clients may only participate in youth development services or participate in them as 
well as receive clinical services. For some clients receiving clinical services, participation in 
youth development activities is not appropriate since they may first need to stabilize their 
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Box 4.3 

Examples of Youth Development Activities Offered by Cll* 

Life and Social Skills: employment preparation courses, a youth internship program, and 
SOAR, a group program improving communication and relationship skill s . 

Literacy and Education: homework help, a children’s library, and a computer lab. Education 
programs include Superstart, a program reinforcing basic literacy and math skills, and reading 
programs such as Stories Abroad and Camp Read-A-Lot. 

Creative Arts: painting, drawing, and ceramics classes. Digital media arts programs teach 
film-making and visual design, while performing arts offerings include theater, music, and 
various dance classes. 

Health and Wellness: workshops in nutrition and healthy eating and cooking classes. Services 
promoting physical activity include exercise classes and sports programs. 


Children’s Institute, Inc. (2015). 


situations through therapy. If appropriate, a therapist might refer a client to a particular youth 
development activity in conjunction with their treatment, for example, to improve social skills 
or overcome anxiety. In other cases, clinicians use youth development activities as a “step 
down,” or transition, for clients completing therapy. In both cases, therapists or case managers 
use youth development activities to enhance the client’s treatment on site and without external 
referrals. “The goal for mental health should never be a lifelong commitment,” one CII therapist 
explained. “Being able to provide services that are not mental health related, but that you can 
still practice social skills, stress management, coping skills, and all that, it’s something that they 
can generalize through their whole life.” Another CII staff member said that “[tjhey need their 
therapy but they need things to build resilience and hope and those kinds of things so that they 
can see beyond what’s happened to them and be kids, have some fun.” Occasionally, youth 
development staff identify participants with mental health needs and arrange clinical services 
for them. 

CII’s wide-ranging youth development activities require a great deal of resources. A 
combination of CII staff, contractors, and volunteers with specific expertise facilitate the various 
activities. The community service department, which organizes youth development activities, 
employs six to eight full time staff members. An additional 25 full-time staff at CII spend a 
portion of their time working on youth development activities. CII also engages approximately 
1,000 volunteers each year, of which about 100 are active at one time. Clinical staff reported 
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some difficulty in finding time to participate in the delivery of youth development activities. 
Many recognized the value of working with their clients in different ways and sometimes 
facilitated activity groups. However, they also said that clinical staff felt pressure to spend their 
time doing “billable” clinical work, which limited how much they could participate in youth 
development activities. 

According to CII records, participants of all ages received youth development services 
during the study period. Table 4.6 shows that participants in SPA 4 attended more classes and 
spent more hours engaged in activities than did participants in SPA 6. This difference was 
expected since CII in SPA 4 had expanded its service offerings by the time of the study period. 
Across all CII operations, nearly 50 percent of participants were under 9 years old and about 40 
percent were 9 to 24 years old. Participants ages 9 to 1 7 years, on average, attended nearly 1 1 
classes for over an hour, totaling 20 hours of participation. The activities targeting 1 8- to 24- 
year-olds included employment supports, a domestic violence program affiliated with Cal- 
WORKs, and parenting classes. The differences in the types of activities may explain the 
varying participation levels among the age groups. 


The Integrated Service Model 

CII coordinates clinical, family support, and youth development services through its Integrated 
Service Model. Designed to ensure children and families receive multiple services tailored to 
their specified needs, it requires staff to weave together a complex set of contractual funding, 
grants, and other sources of revenue. CII developed the Integrated Service Model to eliminate 
the historic silos separating clinical services from the agency’s other activities and to create a 
more fluid communication system to better identify clients’ needs and connect them to all the 
services required to address those needs. 

Staff reported that implementing the Integrated Service Model involved changes in how 
CII organized staff and how staff related to one another and communicated new hires and other 
personnel issues, and how CII funded its services and made them available to clients and their 
families. Staff in different SPAs reported that these changes led to greater communication and 
closer collaboration among programs and departments. The changes also introduced structures 
to convene staff from across programs and departments in order to share important information 
about their work, which in turn prepared staff to better serve clients and their families. CII also 
modified its hiring practices, adding the expectation that new staff members be flexible with 
respect to the locations where or programs in which they work. 

CII strives to provide multiple, coordinated services to clients to address their many 
needs and high-risk factors. It is difficult to assess the progress that CII made in implementing 
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Table 4.6 


Client Demographic Characteristics and Participation in 
Youth Development Services 


Characteristic 

All CII 

SPA 4 

SPA 6 

Participates in anv vouth development services 

Age a (%) 




Under 9 years 

48.9 

43.9 

62.9 

9-17 years 

34.6 

35.8 

32.1 

18-24 years 

2.2 

2.3 

0.7 

25 years or older 

13.0 

17.7 

4.0 

Gender (%) 




Male 

45.8 

41.9 

56.1 

Race/Ethnicity (%) 




Hispanic/Latino 

79.9 

91.1 

70.0 

White 

12.3 

3.3 

28.5 

Black 

3.5 

1.0 

0.6 

Asian 

2.6 

3.8 

0.3 

Other 

1.5 

0.9 

0.6 

Sample size 

4,013 

2,324 

808 

Averase number of classes attended per participant 




Age 




9-17 years 

10.7 

12.9 

6.6 

18-24 years 

12.3 

14.7 

5.8 

Averase service time per contact 11 (hours) 

Age 




9-17 years 

1.5 

1.3 

1.7 

18-24 years 

0.6 

0.6 

1.4 

Averase service time per participant (hours) 

Age 




9-17 years 

20.0 

25.0 

11.6 

18-24 years 

22.7 

34.5 

8.3 

Sample size 

1,477 

886 

265 


(continued) 
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Table 4.6 (continued) 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, through 
June 30, 2013, and received services from January 1, 2012, through December 31, 
2013. 

This analysis excludes one-day youth development events. 
a Age category percentages may not sum to 100 percent because of missing data. 
b Average service time per contact (hours) was calculated using a different unit of 
analysis from the unit of analysis used to calculate other measures in the table. For 
this measure, data are presented as an average per contact, while other measures are 
presented as averages per participant. 


the Integrated Service Model because the research team did not have access to the data on client 
risk factors, which are necessary to determine whether clients receiving only one service would 
have benefited from other services. The team instead analyzed data on clients receiving multiple 
services, since the extent to which clients participated in multiple services serves as a proxy for 
measuring CII’s progress in connecting clients to multiple, coordinated services. As shown in 
Table 4.7, about 60 percent of clients received only one type of service during the analysis 
period, most of whom participated in a youth development activity. Only 9 percent of clients 
received only clinical services during the study period. Of the nearly 40 percent of clients 
receiving multiple services, most received clinical services and at least one other type of service. 
Of these clients, over 40 percent received clinical and family support services during the study 
period, which is unsurprising since many of CD’s clinical services integrate family support 
services. For example, parents or guardians in the Full Service Partnership program automati- 
cally receive services from a parent partner. Due to limitations of Oil’s information manage- 
ment system at the time of the study, it was not possible to determine what percentage of clients 
received family support services as part of their clinical treatment, or as part of a separate 
program requiring a separate referral. More telling of CII’s progress in engaging clients in 
multiple services, however, is that more than 50 percent of clients receiving clinical services 
were also receiving family support and youth development services. 15 


15 Table 4.7 does not distinguish participation by SPA. If it did, it would be expected that SPA 4 would 
have higher participation levels because of the significant emphasis on the Integrated Service Model in that 
location. CII staff did not emphasize it as much in SPA 6 and even less so in SPA 8. 
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Table 4.7 


Client Participation in CII Services 


Characteristic All CII 

Participates in one service 3 (%) 60.8 

Clinical services only 8.9 

Family support services only 27.9 

Youth development services only 63.2 

Participates in multiple services 3 ’ 1 * (%) 38.7 

Clinical and family support services only 42.6 

Clinical and youth development services only 3.5 

Family support and youth development services only 1.1 

Clinical, family support, and youth development services 53.9 


Sample size 6,566 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, 
through June 30, 2013, and received services from January 1, 2012, through 
December 31, 2013. 

This analysis excludes one-day youth development events. 
a These measures represent the percentage of clients among all those who 
belong to this subset of CII clients. 

b This characteristic includes only clients who participated in clinical services 
and at least one other type of service. It does not include clients who 
participated in multiple services but not clinical services; it excluded 29 clients 
who only participated in family support and youth development services only. 


Though it appears as if CII successfully implemented the Integrated Service Model, it 
was not entirely possible to assess the effectiveness of the integrated services. Hypothesizing 
that participation in multiple services would improve the clients’ outcomes, the research team 
analyzed pre- and post-Youth Outcome Questionnaire scores for the limited number of clients 
who received TF-CBT or FFT as well as family support services, youth development services, 
or both. 16 Results of this analysis did not support the hypotheses; the research team found no 


16 The team chose TF-CBT and FFT because they were the services for which data were most likely to 
have been collected or available. 
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correlation. However, data on outcomes were not available for all clients, making it difficult to 
fully interpret this finding. 17 Appendix A describes the limitations of this analysis in more detail. 

Staff described several benefits of the Integrated Service Model, particularly the value 
of combining clinical services with youth development activities. All clinicians who responded 
to the staff survey thought that receiving youth development services in conjunction with 
clinical services improved client outcomes to some degree compared with receiving only 
clinical treatment; nearly 50 percent thought that it improved client outcomes a “great deal.” 
One CII therapist described the changes in a child who attended weekly therapy sessions but 
would not talk. The therapist referred the client to the performing arts program, where the client 
began writing poetry, developed an interest in media, experimented with filmmaking, and wrote 
part of a play. The therapist went to one of the client’s performances and it thrilled her. She 
described the ability of these enrichment activities to complement therapy as “magical.” 
Another therapist described similar positive effects on some clients who enrolled in CII’s art 
class. “A few of the kids who are in it have really opened up because I think it’s [art class] 
accessed a part of them that’s really allowed them to express themselves,” the therapist ex- 
plained. “And so I think that’s helped them within their regular [therapy] sessions to really just 
start verbalizing their feelings more and just be more comfortable with who they are.” 

A more comprehensive assessment of the impact of the Integrated Service Model on 
clinical services would require more data than are currently available and a different study 
design. For instance, such an assessment would need more years of data than analyzed in this 
study to capture the full duration of the clinical and youth development services clients may 
receive. The assessment might also include pre- and post-tests for clients participating in youth 
development activities. CII is currently experimenting with these tests, which make them likely 
components of any such assessment in the future. 


Conclusion 

CII provided clients with a variety of services during the study period. The largest proportion of 
clients participated in youth development activities; a smaller proportion received clinical 
services. Therapists and other CII staff have on the whole bought into the Integrated Service 
Model, which CII has implemented as evidenced in part by the analysis of clients receiving 
multiple services. It is notable that many clients in clinical services also receive other services at 


n The research team also analyzed the relationship between the number of treatment sessions and signifi- 
cant improvements in outcome test scores; results similarly did not support the hypothesis that more treatment 
leads to more improvement. 
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CII. Delivering services according to the Integrated Service Model is not without its challenges, 
however. Perhaps the greatest challenge is providing multiple, coordinate services in the 
current, complex funding environment. And while the SIF grant allowed CII to expand its 
programming, as the grant comes to an end, CII must find new ways to sustain its activities. 
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Chapter 5 

A Closer Look at Two Evidence-Based Practices 


Children’s Institute, Inc. (CII), has pioneered the use of evidence-based practices. 1 In 2012, 65 
percent of CII clinicians were trained in more than a dozen different evidence-based practices 
(Table 4.2). Each evidence-based practice has its own requirements and regulations for adoption 
and implementation. The developers of many evidence-based practices make materials readily 
available to the general public at little or no cost and do not require specific trainings or moni- 
toring. 2 Other developers, such as those behind Functional Family Therapy (FFT), require 
certification and continual monitoring to implement and use the treatment. CII also uses a 
number of “evidence-informed” practices that, while not rigorously tested, are based on varying 
levels of evidence; 3 almost one-quarter of clients receiving CII’s clinical services engage in an 
evidence-infonned practice. 

About one-third of CII’s clients receiving clinical mental health services engaged in an 
evidence-based practice (Table 4.1). 4 And nearly half of clients receiving clinical services 
engaged in evidence-based or evidence-infonned practices. 5 As Table 5.1 shows, 57 percent of 
the clients engaged in an evidence-based practice were under 9 years of age. One reason for this 
high number is that more evidence-based practices are available to this age group. Another 41 
percent of clients engaged in an evidence-based practice were 9 to 24 years old; this group 
represents just over one-third of all 9- to 24-year-olds receiving clinical services at CII (not 
shown). The percentage of clients receiving evidence-based treatment at CII is higher than 


'Evidence-based practices include the following treatments and programs: Cognitive Behavioral Interven- 
tion for Trauma in Schools, Child Parent Psychotherapy, Functional Family Therapy (FFT), Incredible Years, 
Managing and Adapting Practice, Multidimensional Treatment Foster Care, Parent-Child Interactive Therapy, 
Reflective Parenting Program, Trauma-Focused Cognitive Behavioral Therapy (TF-CBT), and Trauma 
Systems Therapy-Substance Abuse. 

2 For example, Cognitive Behavioral Intervention for Trauma in Schools makes all necessary materials 
available for free on its website. 

3 Evidence-informed practices include the following treatments and programs: Domestic Violence Treat- 
ment Groups, Project Fatherhood, Wraparound services, Youth with Sexual Behavior Problems programs, and 
social skills and parent support groups. Some evidence informs these practices but not as much as evidence- 
based practices have accumulated. 

4 The remaining young people received other therapies not classified as evidence based. ITowever, without 
knowing more about client risk and medical necessity, it is not possible to say whether this saturation is 
appropriate or if it falls short. 

5 After accounting for the overlap of those clients engaged in both evidence-based and evidence-informed 
practices during the study period, 47 percent of clinical clients participated in either an evidence-based or 
evidence-informed practice (not shown). 
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Table 5.1 


Demographic Characteristics of CII Clients Engaged in 
Evidence-Based Practices 3 


Characteristic 

All CII 

SPA 4 

SPA 6 

Engaged in any Evidence-Based Practice 11 (%) 




Age 




Under 9 years 

56.9 

57.7 

62.7 

9-17 years 

40.1 

38.6 

35.9 

18-24 years 

0.6 

0.0 

0.0 

25 years or older 

1.1 

1.0 

0.5 

Gender (%) 




Male 

51.4 

54.3 

50.7 

Race/Ethnicity (%) 




Hispanic/Latino 

78.1 

94.7 

69.6 

White 

13.8 

2.1 

28.1 

Black 

5.5 

1.3 

1.4 

Asian 

0.4 

0.3 

0.0 

Other 

2.2 

1.6 

0.9 

Sample size 

938 

371 

215 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, 
through June 30, 2013, and received services from January 1, 2012, through 
December 31, 2013. 

“Evidence-based practices include the following treatments and programs: 
Cognitive Behavioral Intervention for Trauma in Schools, Child Parent 
Psychotherapy, Functional Family Therapy, Incredible Years, Managing and 
Adapting Practice, Multidimensional Treatment Foster Care, Parent-Child 
Interactive Therapy, Reflective Parenting Program, Trauma-Focused 
Cognitive Behavioral Therapy, and Trauma Systems Therapy for Substance 
Abuse. 

b Age category percentages may not sum to 100 percent because of missing 
data. 
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average nationwide. According to one estimate, only 2 percent of youth receiving mental health 
services through California’s county-level departments of mental health received an evidence- 
based treatment. 6 This estimate used a narrow definition of evidence-based practice and did not 
include Trauma-Focused Cognitive Behavioral Therapy (TF-CBT), the most popular evidence- 
based practice at CII. Still, even if this estimate is low, CII engages a notably high proportion of 
clients in evidence-based practices. 

Therapists reported some common challenges to implementing evidence-based practic- 
es in general. Implementing any new practice involves a learning curve and requires additional 
training and supervision, which can be time consuming. CII provides training to clinicians and 
support staff for each evidence-based practice they deliver, which may be offered on site or at 
another agency or organization. Another challenge is the cost of training. Training staff in FFT, 
for example, is particularly expensive, and managers take its cost into account when deciding 
what staff members they train in the practice. As one executive staff member explained, “[n]ow 
we know it takes a certain kind of person that has that commitment to FFT . . . Before we spend 
the money training them... we want a commitment that they’re the right person and that they 
will stay and that this is what they’re passionate about.” 

This chapter describes two of CII’s most used evidence-based practices in more detail: 
FFT and TF-CBT. It also explains the requirements for implementing each evidence-based 
practice and how CII staff delivered them during the study period. 7 The chapter presents the 
dosage of each treatment that CII clients received, since dosage is an important indicator of 
whether or not the practice was delivered with fidelity to the model. Lastly, the chapter summa- 
rizes findings from a fidelity study of CII’s TF-CBT services. 8 


Functional Family Therapy (FFT) 

FFT is a family-based treatment for youth ages 11 to 18 years with a range of behavioral 
problems, such as conduct disorder or substance abuse. The youth have often come in contact 
with the juvenile justice, mental health, or child welfare systems. Grounded in the theory that a 
child’s behavioral problems are a symptom of family dysfunction, FFT seeks to create function- 
al family relationships by improving communication and relationships among family members, 
increasing a sense of mutual support, and decreasing intense negativity and dysfunctional 
patterns of behavior. 9 FFT uses both behavioral and cognitive behavioral interventions to 


technical Assistance Collaborative and Human Services Research Institute (2012). 

7 The analysis presented here is limited to clients who enrolled from January 1, 2012, through June 30, 
2013, and received services from January 1, 2012, through December 31, 2013. 

S A Technical Resource for this report presents the full study and is available on the MDRC website. 
9 Henggeler and Schoenwald (2011). 
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develop the strengths of each family member, which in turn gives the family a foundation on 
which to build functional relationships. Randomized controlled trials and quasi-experimental 
studies have demonstrated FFT’s effectiveness in reducing antisocial behavior, recidivism 
among juvenile offenders, and criminal offenses committed by siblings. 10 

The therapist delivers FFT in five phases, which Box 5.1 describes in detail. The thera- 
pist also assesses the youth and family on an ongoing a basis, beginning with the Youth Out- 
come Questionnaire that the youth and parent or guardian complete at intake. 11 During the first 
phases of treatment, the therapist concentrates on engaging the client and family members and 
motivating them to change their behaviors. Next, the therapist works with individual family 
members as well as with the family as a group to change behaviors. Finally, the family focuses 
on sustaining the changes it has made and preventing relapses. 12 At the end of FFT, the therapist 
administers the Youth Outcome Questionnaire again (and potentially other assessments) to 
assess treatment progress and determine if the youth or family should be referred to another CII 
program or activity. 

FFT is a relatively short-term treatment, on average ranging from 8 to 12 one-hour ses- 
sions for mild situations to as many as 30 sessions for more difficult situations. 13 Sessions 
typically take place on a weekly basis over a three- to four-month period. Therapists will 
discharge families from treatment if they cannot make contact for one month. FFT has been 
conducted in clinical, outpatient, and home-based settings. Therapists at CII reported that they 
hold sessions at school, home, or the CII office, depending on what is most convenient for the 
family. Because the whole family participates in FFT, therapists usually hold sessions in the 
evening. “We meet the family where they are at,” a slogan therapists at CII often repeated. 
Typically, therapists conduct FFT as part of a team-based program such as Wraparound services 
or the Full Service Partnership program. During sessions, therapists engage the whole family 
and often use interactive, hands-on activities such as board and card games and improvisation. 
Box 5.2 presents one client’s experience with FFT. 

Adopting FFT 

Functional Family Therapy LLC (FFT LLC) developed FFT and is responsible for dis- 
seminating it and ensuring model fidelity. It requires that all providers throughout the world 


10 Waldron and Turner (2008); Parsons and Alexander (1973). 

"The child and parent or guardian complete different versions of the questionnaire. For a description of 
the questionnaires, see www.oqmeasures.com. 

12 Sexton and Alexander (2000). 

13 The Functional Family Therapy, LLC (FFT LLC) website states that the average treatment consists of 12 
to 14 sessions spread over three to five months, which varies slightly from what Cll staff reported. 
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Box 5.1 

Phases of FFT* 

• Engagement: The goal of this phase is to build trust in the family-therapist relationship, 
so that family members are invested in the treatment process. The therapist demonstrates a 
commitment to be supportive, responsive, and respectful of the family by being available 
to address the family’s needs and by maintaining contact with all family members. 

• Motivation: In this phase, the therapist works to increase a sense of hope and decrease 
hostility within the family by suggesting alternatives to maladaptive family dynamics. The 
therapist maintains a strengths-based approach to improving relationships, exploring con- 
sequences of positive and negative interactions and orienting family members toward a 
positive outlook. 

• Relational Assessment: In this phase, the therapist identifies interaction patterns, focus- 
ing on each dyad within the family to understand the motivations or relational “functions” 
for each individual’s behaviors. Through observation and questioning, the therapist devel- 
ops a relational perspective to inform later phases of treatment. 

• Behavior Change: The therapist focuses on skill building to address behavior that 
prompted the referral to FFT. The therapist teaches communication skills using tasks, 
technical aids, and modeling strategies, while maintaining a culturally appropriate, con- 
text-sensitive, and tailored approach to address the family’s needs. 

• Generalization: The goal of this phase is to build on progress made in the previous phase 
by applying newly developed skills to multiple contexts. Family members learn to address 
future challenges and plan for relapse prevention. This phase focuses on incorporating 
community systems (including education and justice systems) into the treatment process 
and helping the family establish link s in the community. 


*Functional Family Therapy, LLC (2015). 


complete a rigorous training and certification process and regularly monitors providers for 
fidelity assurance. For instance, FFT LLC operates the Clinical Service System, a computer- 
based tracking and assessment system in which providers must log infonnation about their 
cases and progress in FFT. 14 


14 At CII, therapists must also log information about their cases and progress in FFT in Cll’s management 
information system. 
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Box 5.2 

Maya’s Story 

The Los Angeles County Department of Children and Family Services referred Maya to Cll 
when she was 7 years old, shortly after her biological father reunited with her and removed 
from her mother’s care. Maya and her father had difficulties adjusting to the change and she 
began exhibiting behavioral problems. 

Staff at Cll recommended Maya for Incredible Years, an evidence-based group intervention 
that aims to strengthen parent-child relationships. After completing the program, Maya 
showed a notable improvement in her behavior. Maya continued to receive services at Cll for 
another two years, meeting weekly with a therapist to continue to work on her treatment goals, 
which included fewer outbursts toward her father. 

At age 9, two months after reaching her treatment goals and being discharged by her therapist, 
Maya returned to Cll. Prompted by an incident at school that suggested suicidal intent, the Los 
Angeles County Department of Children and Family Services referred Maya for additional 
mental health services. Maya was referred to Functional Family Therapy (FFT), with treat- 
ment goals centered on reducing Maya’s outbursts of anger, which were often directed toward 
her father. Maya had not reached the minimum age required by the FFT model, but because of 
her maturity level, prior participation in CIl’s services, and removal from her mother’s care, 
Cll staff considered the model an appropriate treatment for her. In making the determination, 
Cll staff sought guidance from FFT LLC. 

Maya’s FFT therapist visited her and her father at their home, and began treatment by engag- 
ing and motivating them with discussions of trauma and focusing on emotional development 
and psychoeducation. The therapist explained that Maya and her father had different ways of 
thinking, which sometimes led to conflicts between the two. Through games and discussion, 
she helped them begin to identify barriers in their communication. 

After about two months, the therapist and family proceeded to the behavior change phase. The 
therapist recommended various skills and strategies for Maya and her father with which to 
work together to improve their interactions, including paying each other more compliments, 
creating a chore chart, and organizing family dinners and movie nights. After another two 
months, they were ready to generalize the skills learned, helping Maya learn to apply them in 
different settings, and plan appropriate responses to her triggers of aggressive behavior. 

By the end of her treatment, Maya stated that she felt close to her father and enjoyed discuss- 
ing her feelings with him. On her FFT discharge form, Maya indicated that before FFT her 
relationship with her family had been “very bad;” afterward, however, she considered it 
“mostly good.” Using the same questionnaire, her father also noted an improvement, noting 
that they had progressed from “so-so” to “very good.” On their fomis, they both also indicated 
that they were hopeful or confident that their family dynamic would improve. 
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Organizations wishing to implement FFT must complete a thorough certification pro- 
cess. Once an organization applies to and is approved by FFT LLC to implement FFT, the 
organization enters a contract with FFT LLC and prepares for FFT training, which includes 
selecting or hiring therapists and purchasing necessary materials and assessments. The first of 
three phases lasts 12 to 18 months and begins with an orientation to FFT and its data collection 
requirements. Therapists must log data about their FFT clients, sessions, and progress into the 
Clinical Service System; the organization and FFT LLC review this infonnation. After orienta- 
tion, therapists receive clinical training in FFT from certified trainers. Once they successfully 
complete training, therapists begin using the practice to treat clients under the supervision of an 
FFT consultant, with whom the therapists have weekly teleconferences to review their cases and 
monitor their fidelity to the model. In California, organizations implementing FFT enter a 
contract with the California Institute for Behavioral Health Solutions (formerly California 
Institute of Mental Health), which provides ongoing supervision, training, and support to all 
FFT providers. 

In the second phase, which lasts one year, the organization takes steps to deliver FFT 
independently. The organization’s FFT supervisor attends training and subsequently takes over 
the weekly supervision of therapists from the FFT consultant or contractor. The supervisor 
instead meets with the consultant or contractor every other week. The supervisor also reviews 
the infonnation that therapists log into the Clinical Service System each week and gives them a 
fidelity score every quarter. The California Institute for Behavioral Health Solutions continues 
to provide ongoing supervision to providers in the state, but the organization assumes more 
responsibility for ensuring model fidelity. 

In the third and final phase, FFT LLC gives the organization more independence, and 
FFT consultants or contractors only meet with supervisors once a month. The organization can 
also renew its contract with FFT LLC for a one-year term in this phase. As part of the contract, 
FFT LLC provides continuing education and training. 

FFT Infrastructure at CM 

CII manages two teams of FFT therapists. 15 The therapists work within programs, such 
as Wraparound services and the Full Service Partnership program, or provide standalone 
therapy. They serve families referred to them from schools, the Los Angeles County Depart- 
ment of Children and Family Services, or other entities. Ideally, therapists provide only FFT 
and no other therapies given the model’s intense demands. Therapists must attend weekly two- 


15 At one time, CII managed three teams. 
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hour team supervision meetings, during which two therapists each week present cases that the 
team evaluates together. 16 Therapists also receive individual supervision as needed. 

Executive staff at CII and FFT LLC and others overseeing FFT tend to agree that it 
takes a certain kind of person to implement and deliver FFT successfully. For example, one 
executive staff member at CII stated that the organization had learned to choose therapists who 
could work nights and occasional weekends since the therapy requires all family members to 
participate, including those who work regular business hours. Experience also taught CII that 
therapists must be 100 percent devoted to FFT to optimize the model. In other words, they 
cannot use other evidence-based practices or offer general therapy. “I think that out of all the 
EBPs [evidence-based practices] that we struggled the most with FFT,” explained an executive 
staff member. “I think the treatment is very intense. The kids are involved often in the legal 
system and they’re very high needs. Because you need to work with the family, the therapists 
that were doing FFT, all of their appointments were at night.” 

Implementation of FFT 

Analysis of FFT data from the CII management infonnation system indicates CII im- 
plemented the model with fidelity with respect to treatment dosage. 17 As Table 5.2 shows, 
families on average completed about 15 sessions over the course of five months. 18 More than 
half of all clients attended 13 or more sessions. This number exceeded the expected 8 to 12 
sessions and may be explained by therapists extending treatment to address complex trauma, 
multiple behavioral issues, or family crises. About one-quarter of clients receiving FFT attended 
five or fewer sessions; it can be assumed that these clients did not complete the treatment. A 
client may have stopped treatment for a number of reasons. For instance, the therapist may have 


16 The FFT therapists at CII are currently participating in the Building Outcomes with Observation-Based 
Supervision: An FFT Effectiveness Trial (BOOST), a study of FFT supervision practices within the California 
Institute for Behavioral Health Solutions system. They were randomly assigned to two groups: a “supervision 
as usual” group (the control group) and a BOOST group which receives additional supervision from FFT LLC. 
See clinicaltrials.go v/show/NCTO 1614015. 

17 Fidelity consists of a number of dimensions, and researchers focus on different aspects of fidelity. These 
aspects include: targeting the specific population, providing the treatment with the appropriate strength (such as 
dosage and frequency of sessions), and adhering to the specified treatment model (therapist adherence). Due to 
the limited data, this study only assessed the fidelity of FFT implementation with respect to specific aspects of 
treatment strength. 

bn comparison, the California Institute for Behavioral Health Solutions collects program and outcome 
data on FFT clients for agencies in California. In the most recent available data, the average number of sessions 
was 14. These data represent clients who participated in FFT through the California state-sponsored FFT 
Community Development Teams through the end of January 2009. Each dashboard report reflects only those 
sites that submitted data at the specified data submission date. See California Institute for Behavioral Health 
Solutions (2009). 


64 



Table 5.2 


Client Participation in Functional Family Therapy 3 


Characteristics 

All CII 

SPA 4 

SPA 6 

Participated in any session (%) 




Attending 1-2 sessions 

16.5 

15.0 

23.3 

Attending 3-5 sessions 

8.2 

10.0 

16.7 

Attending 6-12 sessions 

17.7 

15.0 

16.7 

Attending 13-20 sessions 

28.2 

50.0 

23.3 

Attending 21 or more sessions 

29.4 

10.0 

20.0 

Average number of sessions 

15.3 

13.4 

11.7 

Average total session time b (hours) 

33.1 

26.6 

26.0 

Duration of sessions (months) 

4.9 

4.4 

4.0 

Sample size 

85 

20 

30 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, through 
June 30, 2013, and received services from January 1, 2012, through December 31, 
2013. 

a To minimize the influence of data entry errors, participation in Functional Family 
Therapy was determined by using only those records associated with a trained 
Functional Family Therapy clinician. The data were further limited to only those 
records with procedure codes that should be attributed to the therapy. 

b Total session time includes time spent by the therapist conducting the session, as 
well as traveling and completing documentation required by the Los Angeles County 
Department of Mental Health, CII, and the model's developers. As a result, reported 
session times and calculated averages are higher than expected of typical Functional 
Family Therapy sessions. 


determined FFT was inappropriate for a family after a few sessions or because the child was 
placed in foster care or the family situation had changed in some other significant way. The 
data, however, did not indicate the reasons why clients left treatment. The average total FFT 
session time at CII aligned with the model’s expected 8 to 12 hours, after the research team 
subtracted the travel time (often to the family’s home) and the time taken to complete the 
required documentation for the funder and FFT LLC that therapists included in the session time 
they recorded. There were some differences in dosage between SPA 4 and SPA 6, although it is 
unclear what drove these differences. While the dosage that therapists at CII delivered appears 
in line with the recommended dosage, this study could not assess fidelity to FFT within these 
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treatment sessions. One therapist explained that ensuring fidelity with respect to dosage was 
easier when providing FFT as part of Wraparound services or the Full Service Partnership 
program because the therapist could rely on case managers and parent partners to address 
nonclinical needs. When delivering it as a stand-alone service, a family with nonclinical needs 
might sidetrack the therapist from treatment progress and thereby extend the length of the 
treatment (or dosage). 


Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) 

TF-CBT incorporates principles from cognitive, behavioral, and family therapy. Unlike FFT, 
which requires the participation of the whole family, TF-CBT primarily focuses on the child 
and only involves the parents or guardian at specific points in therapy. Cognitive therapy aims 
to change behavior by addressing an individual’s thoughts and perceptions, especially those that 
create distorted views; behavioral therapy seeks to modify habitual responses as a means to 
identify triggers; and family therapy examines patterns of interactions among family members 
to alleviate problems. 19 With respect to family therapy, TF-CBT requires that the parent or 
guardian who did not cause the trauma attend separate sessions to leam how to support the 
child, improve parenting practices, and reduce emotional distress about the child’s trauma. 20 In 
TF-CBT, the therapist also meets with the child and the parent or guardian jointly to help them 
gradually confront and process and the trauma using cognitive behavioral techniques, such as 
teaching them how to disconnect negative emotions from trauma-related thoughts. The therapist 
guides the child in preparing the “trauma narrative,” who then shares it with the parent or 
guardian verbally, in writing, or artistically. In TF-CBT, trauma-affected clients gradually 
confront and process traumatic experiences, which in turn diminishes the prevalence of anxiety 
and avoidant responses, while the parents or guardians leam to therapeutically support their 
children. 21 

Originally developed to treat sexually abused children, 22 TF-CBT has proven to be ef- 
fective in treating children ages 3 to 1 8 years with significant behavioral problems or emotional 
difficulties, such as post-traumatic stress disorder (PTSD), depression, anxiety, and feelings of 
shame and self-blame associated with exposure to traumatic events. 23 Examples of traumatic 


19 Barker (2003). 

20 Cohen, Mannarino, Murray, and Igelman (2006). 

21 Deblinger, Lippmann, and Steer (1996); Deblinger and Heflin (1996). 

22 Cohen, Deblinger, and Mannarino (2004). 

~ 3 A number of studies have demonstrated the effectiveness of TF-CBT in helping children overcome 
symptoms following a traumatic event. See Deblinger, Lippman and Steer (1996); Cohen and Mannarino 
(1996a, b, 1998a, b); Deblinger, Stauffer, and Steer (2001); Cohen, Deblinger, Mannarino, and Steer (2004). 
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events include physical or sexual abuse, community violence, domestic violence, or accidents. 24 
TF-CBT has also proven to be effective in reducing the symptoms of depression and anxiety in 
parents or guardians of trauma-affected children, as well as in improving their parenting 
practices. 25 Box 5.3 presents one CII client’s experience before and after TF-CBT. 

To determine if TF-CBT is an appropriate treatment for a client, therapists at CII screen 
the client using the University of California Los Angeles (UCLA) PTSD-Reaction Index. Los 
Angeles County Department of Mental Health — the primary funder of TF-CBT services — 
also requires that clients and their parents or guardians complete the Youth Outcome Question- 
naire before starting treatment. TF-CBT’s consists of the following central components: 
psychoeducation and parenting skills, relaxation, affective modulation skills, cognitive coping 
and processing, trauma narration and processing, in vivo desensitization, conjoint child-parent 
session, and enhancing safety skills (referred to as PRACTICE). 26 Box 5.4 describes each 
component in detail. “The nice thing about TF-CBT is that it provides a framework,” explained 
one therapist. “You have the luxury of having an assessment period to where you can identify 
where the client sees the greatest need and provide psychoeducation. It helps them understand 
their situation from a more removed perspective and then gradually deal with their personal 
trauma. It gradually becomes more intimate and challenging.” 

To deliver TF-CBT with fidelity, the therapist must use each of the PRACTICE com- 
ponents, and in the correct order, during the treatment within a “reasonable” time period. 27 The 
model does allow the therapists to skip a few components in particular situations. For example, 
the therapist does not need to deliver the in vivo exposure component if the child is not experi- 
encing trauma-related triggers. The therapist may also omit the trauma narrative component if 
an appropriate parent or guardian is unavailable. Therapists have flexibility to tailor sessions to 
fit the specific needs of the client. Therapists can use different techniques such as play or art 
therapy. Furthermore, the therapist may need to modify TF-CBT to accommodate a client’s 
cultural background. For example, a therapist described a sexual abuse case, in which the client 
expressed cognitive distortion associated with shame that indicated the client’s religious 


"'According to the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Dis- 
orders (DSM-IV-TR), a traumatic event involves (1) actual or threatened death or serious injury, or a threat to 
one’s physical integrity, or witnessing an event that involves death, injury, or a threat to physical integrity of 
another person; or learning about unexpected or violent death, serious harm, or threat of death or injury 
experienced by a family member or other close associate; and (2) the person’s response to that event must 
involve intense fear, helplessness, or horror. See American Psychiatric Association (2000). 

25 Cohen, Mannarino, Murray, and Igelman (2006); Cohen and Mannarino (1997); Deb linger, Steer, and 
Lippmann (1999). 

26 Cohen, Mannarino, Murray, and Igelman (2006). 

27 The TF-CBT developers do not explicitly set parameters for a “reasonable” time period. 
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Box 5.3 

Sofia’s Story 

In March 2013, 10-year-old Sofia visited CII for treatment. Her mother was concerned be- 
cause Sofia cried frequently and was very sensitive and shy. She was also frequently irritable 
and had difficulty getting along with her two younger siblings. 

Sofia’s mother believed that her daughter’s problems began four years earlier, soon after her 
separation from Sofia’s father. Then, two years after the separation, Sofia’s baby sister had an 
accident while in her care, resulting in a head injury. The accident prompted an investigation 
by the Los Angeles County Department of Children and Family Services and ultimately 
resulted in the removal of Sofia and her two sisters from their mother’s care. Sofia’s experi- 
ence of the accident and its consequences traumatized her and led her to develop strong 
feelings of guilt and regret. 

Following the accident, Sofia lived with her father for eight months. After her mother regained 
custody, Sofia began experiencing anxiety, especially when separated from her mother. Sofia 
felt guilty about the accident and its consequences. 

As part of CII’s initial assessment process, Sofia met with a therapist who helped her complete 
an assessment of her past exposure to trauma. Sofia’s mother completed a parent version of the 
assessment, as well as another assessment designed to identify any other behavioral problems. 

After Sofia’s initial assessment, she began in-home TF-CBT to reduce the frequency of the 
post-traumatic stress disorder symptoms that led her mother to seek help, including irritability 
and sensitivity. Sofia’s therapist, Marissa, began treatment with sessions devoted to the 
psychoeducation component of TF-CBT, teaching Sofia about accidents. She explained to 
Sofia the difference between regret and fault and spoke with her about how children may feel 
when involved in accidents. During these sessions, Sofia also learned and practiced stress 
management techniques such as controlled breathing. 

With Marissa’s guidance, Sofia gradually progressed to the affect identification component of 
TF-CBT. Sofia spoke about her sister’s accident, and identified the resulting emotions she 
experienced. During their seventh session, Marissa introduced the trauma narrative compo- 
nent, working with Sofia to develop an account of the accident and helping her feel comforta- 
ble discussing it. By the tenth session, Sofia was able to practice sharing her narrative, with 
Marissa playing the role of her mother. 

Throughout the course of treatment, Sofia’s mother participated in separate sessions with 
Marissa, where she learned about Sofia’s progress as well as strategies to support her daughter 
and reinforce her growth. She also participated in a joint session, in which Sofia shared the 
trauma narrative with her. Coached by Marissa, Sofia’s mother was able to positively reinforce 
and praise her daughter for sharing her narrative. She noted that Sofia seemed calmer and 
more confident, and Sofia agreed that she felt more relaxed overall. 
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Box 5.4 

TF-CBT Components* 

Psychoeducation and Parenting Skills: Educating child and parent about typical reactions to 
trauma. Normalizing symptoms and reducing self-blame, using games, videos, and discussion. 
Parents receive coaching in effective behavior management strategies and communication. 

Relaxation: Teaching children relaxation techniques including focused breathing, mindfulness, 
progressive muscle relaxation, and visual imagery. 

Affective modulation skills: Helping children to identify and understand their emotions. 
Discussing feelings associated with their trauma and developing strategies to regulate emo- 
tions. 

Cognitive coping and processing: Revisiting the role of thoughts in driving emotions, and 
helping children identify harmful or inaccurate thoughts. Teaching children that thoughts can 
be changed and helping to generate more accurate and positive thoughts. 

Trauma narration and processing: Providing gradual exposure to trauma-related memories, by 
recounting the experience. Helping to identify inaccurate trauma-related thoughts and replac- 
ing them with more accurate thoughts about the traumatic event, one’s self, and others. 

In vivo desensitization: Reducing avoidance that disrupts daily functioning by teaching 
children to distinguish trauma reminders from actual danger. Gradually exposing children to 
triggers and helping children practice regulating their emotions and reactions. 

Conjoint child-parent session: Providing an opportunity for supportive communication be- 
tween child and parent or guardian. Children share their trauma narrative with their parents or 
guardians and parents or guardians offer praise and encouragement. Children prepare for the 
conjoint session sharing the full trauma narrative with the therapist over several prior sessions. 

Enhancing safety skills: Creating a written safety plan that identifies risks or triggers and 
provides prepared responses to help children feel safer after completion of treatment. Teaching 
skill s for use in dangerous situations that may arise in the future. 


*Cohen, Mannarino, Murray, and Igelman (2006). 


upbringing, namely “I’m not a virgin anymore and am going to hell.” The therapist adjusted 
TF-CBT by connecting the client with a priest, who explained that virginity is not something 
one person can take from another through abuse. 

Therapists provide TF-CBT in the client’s home, at school, in the CII office, or wherev- 
er else the client is comfortable. Therapists do not typically deliver the treatment as part of a 
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team-based program such as Wraparound services or the Full Service Partnership program. 
Therapists conduct 60- to 90-minute sessions once a week for 12 to 20 weeks, depending on the 
complexity of the trauma. 28 According to therapists at CII, the relatively short, finite duration of 
the treatment — unlike traditional open-ended therapy, for instance — is attractive to families 
and can promote their engagement. At the end of treatment, therapists again administer the 
UCLA PTSD-Reaction Index and Youth Outcome Questionnaire to gauge clinical improve- 
ment and determine if a client should continue treatment — either TF-CBT or another practice. 

Adopting TF-CBT 

Implementing and delivering TF-CBT involve fewer and less stringent requirements 
than FFT, and providers have sole responsibility for overseeing and supervising the implemen- 
tation and delivery. The developers of TF-CBT do not regularly monitor the intervention; 
however, at some providers (including CII), therapists must engage in some TF-CBT supervi- 
sion. For instance, therapists and their supervisors use the TF-CBT developers’ Brief Practice 
Checklist to track the progress of each client through the TF-CBT components and to support 
therapist adherence to the model. Although the developers offer a certification in TF-CBT, they 
do not require it to deliver the model. 29 The Los Angeles County Department of Mental Health, 
however, does set some requirements for the TF-CBT providers it funds, including CII, such as 
submitting baseline and outcome data to the California Institute of Behavioral Health Solutions, 
which tracks model fidelity and providers’ performance. 

TF-CBT Infrastructure at CII 

CII requires therapists providing TF-CBT to participate in group supervision. CII oper- 
ates two groups: one for therapists new to TF-CBT, which focuses on problem solving and case 
presentation, another for more experienced therapists, which meets every other week and 
focuses on aspects of cultural modification. 

According to therapists interviewed by the research team, TF-CBT works best in a sta- 
ble home since clients are usually confronting the most difficult events in their lives and need a 
supportive environment. However, given that CII serves low-income urban communities, 
therapists cannot always assume clients live in stable homes. Relatedly, TF-CBT theoretically 
focuses on the trauma-affected client, not the client’s living situation or community. At the end 
of treatment, clients are often still living in the same homes or communities associated with 
their trauma. 


~ 8 Child Sexual Abuse Task Force and Research and Practice Core, National Child Traumatic Stress Net- 
work (2004). 

29 See Trauma-Focused Cognitive Behavioral National Therapist Certification Program website: tfcbt.org. 
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Implementation of TF-CBT 

Table 5.3 presents the analysis of TF-CBT treatment dosage during the study period. 
Findings indicate that the dosage is more or less aligned with model expectations. Clients 
typically participated in TF-CBT for 21 weeks, attending 19 sessions over that time, which 
averages close to one per week. About one-quarter of clients attended the expected 12 to 20 
sessions in that period, whereas 30 percent attended 10 or fewer sessions. These clients may 
have attended a small number of sessions because clients ended their treatment at CII or 
therapists referred them to a different treatment model. In comparison, the most recent data 
from the California Institute for Behavioral Health Solutions, which collects program and 
outcome data on TF-CBT clients in California, indicate that clients on average attended 19.5 
sessions over 26.1 weeks. 30 The average total TF-CBT session time at CII aligned with the 
model’s expected 12 to 30 hours, after the research team subtracted the travel time (often to the 
family’s home) and the time taken to complete the required documentation that therapists 
included in the session time they recorded. There are some differences in dosage between SPA 
4 and SPA 6, although it is unclear what drove these differences. 


TF-CBT Fidelity Study Findings 

The fidelity study of CII’s TF-CBT services used an observational method. (A Tech- 
nical Resource for this report presents the full study and is available on the MDRC website.) CII 
offered new clients assigned to TF-CBT between November 2013 and August 2014 the oppor- 
tunity to participate in the study; 126 clients enrolled in the study. CII distributed study partici- 
pants across 3 1 therapists, who would audio record each treatment sessions, including client- 
only sessions, parent- or guardian-only sessions, and conjoint sessions. During the period 
between November 2013 and February 2015, therapists collected more than 1,009 recordings, 
half of which a research team from the Medical University of South Carolina (MUSC) coded. 
The MUSC team selected recordings evenly across several factors: number of therapists, 
number of clients per therapist, and number of sessions per client. The team coded four clients 
per therapist and weighted the sampled recordings to ensure adequate representation of coded 
sessions across each TF-CBT component. 

The MUSC team used a version of the Therapy Process Observational Coding System 
for Child Psychotherapy designed for TF-CBT (TF-CBT TPOCS-S) to assess therapists’ 


30 These numbers are based on data submitted to California Institute for Behavioral Health Solutions in 
September of 2010 and represent clients who received TF-CBT through the Institute’s TF-CBT Community 
Development Teams through the end of July 2010. Each dashboard report reflects only those sites that 
submitted data at the specified data submission date. See California Institute for Behavioral Health Solutions 
( 2010 ). 
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Table 5.3 


Client Participation in Trauma-Focused Cognitive Behavioral Therapy 3 


Characteristic 

All CII 

SPA 4 

SPA 6 

Participated in any session (%) 




Attending 1-5 sessions 

19.8 

16.9 

11.6 

Attending 6-10 sessions 

9.9 

9.7 

10.5 

Attending 11-15 sessions 

11.9 

6.5 

23.3 

Attending 16-20 sessions 

16.2 

16.9 

22.1 

Attending 21-25 sessions 

14.9 

16.1 

15.1 

Attending more than 26 sessions 

27.4 

33.9 

17.4 

Average number of sessions 

19.3 

21.3 

18.0 

Average total session time b (hours) 

25.5 

23.6 

30.5 

Duration of sessions (weeks) 

21.4 

22.7 

22.1 

Sample size 

303 

124 

86 


SOURCE: CII management information system. 

NOTES: Analysis is limited to clients who enrolled from January 1, 2012, through June 30, 
2013, and received services from January 1, 2012, through December 31, 2013. 

a To minimize the influence of data entry errors, participation in Trauma-Focused 
Cognitive Behavioral Therapy was determined by using only those records associated with 
a trained Trauma-Focused Cognitive Behavioral Therapy clinician. The data were further 
limited to only those records with procedure codes that should be attributed to the therapy. 

b Total session time includes time spent by the therapist conducting the session, as well as 
traveling and completing documentation required by the Los Angeles County Department 
of Mental Health, CII, and the model's developers. As a result, reported session times and 
calculated averages are higher than expected of typical Trauma-Focused Cognitive 
Behavioral Therapy sessions. 


adherence to the model through all of its 12 components. 31 The team then rated their use of and 
fidelity to these components on a seven-point scale. Fidelity was assessed across a course of 
treatment, reflecting that TF-CBT components are intended to be delivered in phases and 
that it is not expected that every session will include all of the components. 


3l The TF-CBT TPOCS-S is an observational coding system that the National Institutes of Mental Health 
developed as part of a multiyear randomized controlled trial studying the effects of clinical supervision 
strategies on how a large sample of community-based therapists in Washington State implemented TF-CBT. 
The tool is not yet publicly available. 
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In addition to measuring therapists’ adherence to the TF-CBT model, the study also 
evaluated the psychometric properties of the TF-CBT Brief Practice Checklist, the self- 
reporting fidelity monitoring tool therapists at CII use. Because observational methods such as 
the TF-CBT TPOCS-S are resource intensive, the research team wanted to assess the value and 
effectiveness of this less costly fidelity monitoring tool. Therapists complete the Brief Practice 
Checklist after each session, noting which components of TF-CBT they implemented in each 
session. TF-CBT supervisors review the checklists during group supervision meetings. The 
fidelity study found that, through the course of treatment, CII clients were more than 50 percent 
likely to receive half of the components. This finding indicates that CII therapists did not deliver 
the model’s full prescribed course of treatment and is in line with previous research. Clients 
were most likely to receive the cognitive coping, relaxation, affective expression and modula- 
tion, psychoeducation, and trauma narrative components. Notably, previous studies of commu- 
nity-based TF-CBT providers found that clients were less likely to receive the trauma narrative 
component. 32 In contrast, nearly all CII clients received this component. Therapists at CII 
implemented the in vivo exposure and parenting components the least frequently. The fact that 
therapists only conduct in vivo exposure when clients are experiencing ongoing trauma-related 
triggers in their everyday enviromnent — a subset of CII clients — may be one reason why 
therapists did not often implement that component. And the low number of parent- or guardian- 
only and conjoint sessions in the study sample may explain the infrequent use of the parenting 
skills component. The TF-CBT model prescribes a relatively equal number of sessions for 
clients and parents or guardians. However, in the study sample, 63 percent of sessions were 
client only, 35 percent were parent or guardian only, and 2 percent were conjoint. The low 
number of parent- or guardian-only sessions was consistent with findings from previous studies 
of community-based TF-CBT providers. Since the study included only observational and self- 
reported data on parent or guardian involvement, it is not possible to determine the reasons for 
this low level of parent or guardian involvement. Therapist may have chosen not to involve the 
parents or guardians at the levels prescribed by the model, or they may have encountered 
challenges in engaging parents or guardians in treatment similar to those therapists in other 
studies faced. 33 

These findings are consistent with what the research team learned from interviews 
with TF-CBT therapists in the implementation study. Therapists gave a number of reasons 
why TF-CBT can be difficult to implement and why they may not implement it with complete 
fidelity. Clients in TF-CBT tend to have a history of multiple traumas and are often experi- 
encing ongoing crises, and therefore therapists might have to conduct a crisis intervention in 
the midst of TF-CBT treatment. Such an intervention may prolong TF-CBT, or sometimes 


32 Allen and Johnson (2012). 

33 Cohen, Mannarino, Kliethermes, and Murray (2012). 
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compel a therapist to temporarily suspend the therapy until the client is stabilized or not as 
high risk. Therapists at CII also described challenges in properly balancing what happens in 
clients’ lives from week to week with making progress in treatment. The trauma narrative 
component, in particular, causes distress and clients often cancel sessions leading up to it. 
These interruptions can derail treatment, and reengaging clients can be difficult. Consistent 
with these accounts, the MUSC team found that nearly all of the sessions they coded included 
crisis or case management. 

The fidelity study aimed in part to assess how model fidelity varied by therapists’ char- 
acteristics. Using TF-CBT TPOCS-S, the MUSC team found that fidelity varied at the client 
level, rather than at the therapist level. In other words, clients treated by the same therapist 
experienced TF-CBT differently. There were no data available to assess the client-level factors 
or characteristics that might explain this variability or the extent to which the modifications 
therapists made to accommodate individual clients were consistent with model fidelity. 

The MUSC team also found that using the Brief Practice Checklist or the observational 
method to assess model fidelity over the course of treatment led to similar findings. However, 
consistent with similar studies, the team found that therapists were more likely to over-report 
adherence to the TF-CBT model on the Brief Practice Checklist. In addition, whereas the TF- 
CBT TPOCS-S attributed variations in fidelity more to clients’ characteristics than those of 
therapists, the study found that the Brief Practice Checklist attributed a higher percentage of the 
variation in fidelity to the therapists’ characteristics. This finding suggests that individual 
therapists respond differently to questions on the checklist. 

The study nonetheless indicates that the Brief Practice Checklist may be a promising 
low-cost tool to monitor fidelity. Observational methods of monitoring fidelity, such as the one 
used in this study, are time and resource intensive and not practical on a large scale for many 
community-based organizations. Therapists and supervisors could use the Brief Practice 
Checklist to monitor the delivery of TF-CBT’s components, and supervisors could use the 
information on the checklist to advise therapists on how to address roadblocks to providing the 
treatment as intended. Additionally, organizations could use data from the checklists to compare 
differing outcomes among clients and identify and assess any patterns. However, there are some 
limitations to using the checklist alone to evaluate fidelity. Therapists in the study tended to 
over-report their use of the therapy’s components relative to findings from the TF-CBT 
TPOCS-S. Additionally, whereas observational methods such as the TF-CBT TPOCS-S can 
measure the extent to which therapists implemented each component, the Brief Practice Check- 
list can only determine whether or not therapists used each component. However, organizations 
could use the Brief Practice Checklist in combination with others tools, such as periodic direct 
or audio-recorded observations, to monitor fidelity more accurately. 
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The study included a fidelity assessment of TF-CBT delivered in Spanish, though small 
sample sizes limited the analysis to single therapy sessions rather than the course of treatment. 
The assessment used the TF-CBT TPOCS-S and found that therapists were more likely to 
deliver parenting skills in sessions conducted in Spanish, perhaps because a greater proportion 
of the Spanish-language sessions in the sample were parent- or guardian-only or conjoint 
sessions relative to the English-language sessions. Therapists in Spanish-language sessions were 
less likely to implement the trauma processing component than those in the English-language 
sessions. 


Conclusion 

CII has been a pioneer in adopting and delivering evidence-based practices. However, it is 
difficult to interpret the saturation of clients engaged in evidence-based practices without 
knowing more about each client’s circumstances, since these practices are not appropriate for all 
clients in every circumstance. 

A sizable proportion of clients receiving CIFs clinical services engage in an evidence- 
based practice. Analysis of data from CIFs management information system indicate that the 
dosage of FFT aligns with model expectations. The dosage of TF-CBT also appears to align 
with model expectations. However, the TF-CBT TPOCS-S found that therapists, on average, 
did not deliver all of the prescribed TF-CBT components. This finding corroborates those from 
previous fidelity studies of community-based TF-CBT providers, underscoring the need to 
improve supports for therapists to help them provide treatment with fidelity. The TF-CBT 
fidelity study also found the Brief Practice Checklist to be a promising low-cost tool for 
monitoring fidelity, though it alone may not be enough to ensure fidelity. 
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Chapter 6 

Discussion 


There is overwhelming evidence that traumatic experiences in childhood can lead to poor 
outcomes in adulthood. While an extensive child welfare field — encompassing policymakers, 
researchers, and practitioners — focuses on improving the life prospects of these children and 
families, the services available to them are often fragmented and uncoordinated. Service 
providers are spread across different agencies, and funding streams often support only specific 
types of care or treatment. Many of the available services also lack evidence of their effective- 
ness, though there has been a push in recent years to increase the use of evidence-based practic- 
es in children’s mental health care. 

This report describes in depth how Children’s Institute, Inc. (CII), tackles these issues. 
Through its Integrated Service Model, CII aims to provide holistic and coordinated support to 
children and families through a combination of clinical and nonclinical services. CII also 
prioritizes delivering evidence-based practices to clients when appropriate. CII serves as an 
important case study that may be useful to other multiservice organizations addressing these 
issues. This chapter summarizes the report’s findings and identifies areas for further research. 


Integrated Service Model 

Through its Integrated Service Model, CII attempts to knit services together to address the 
complex needs of the children and families it serves. Clients receive multiple types of services, 
depending on their needs, throughout their involvement with CII. The Integrated Service Model 
intends not simply to provide clients with multiple types of services, but to eliminate operating 
silos within CII and to create a system that accurately identifies clients’ full range of needs and 
ensures they receive all the support required to address those needs. 

Based on the interviews and available data, MDRC researchers found that CII is 
achieving its goal of engaging clients in multiple types of services to meet their needs. An 
overwhelming majority of clients receiving clinical services were engaged in more than one 
type of service. Staff expressed support for the model and described how it benefited their 
clients. To implement the model, CII had to overcome the challenges of coordinating care 
across a fragmented system. Flexible funding was essential to providing clients with non- 
clinical services. Further research is needed to determine whether CII appropriately targets its 
services — namely, whether all clients who need multiple types of services receive them. 
Additional research is also needed to assess the “value-added” of the different types of services 
CII provides, for instance whether its model improves short- or long-tenn outcomes for clients. 
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At the time of the study, CII was not collecting the necessary data in a way that would have 
allowed the research team to conduct these assessments. 


Evidence-Based Practices 

Though momentum has been building across the country to increase the use of evidence- 
based practices, adopting and delivering them in community-based settings comes with a host 
of challenges. Whether or not an evidence-based practice is effective depends on how the 
community-based provider implements it. In order to transfer efficacy from research to 
practice, providers must implement the practice with fidelity to the treatment model, which 
can be particularly challenging for community-based organizations such as CII. 

This report describes how CII incorporates evidence-based practices into its clinical 
services. Nearly half of clients receiving clinical services at CII engaged in an evidence-based 
or evidence-informed practice, which according to the limited data available is a much higher 
saturation rate than that of other mental health providers. This report identifies the challenges 
that CII encountered implementing evidence-based practices, including securing the additional 
resources required to train and supervise staff and delivering a specified treatment model to 
high-risk clients. Fidelity consists of several components, including adherence to the treatment 
model, dosage, and the quality of implementation. Due to the study design, MDRC researchers 
could only assess particular aspects of fidelity for each of the two evidence-based practices they 
examined. For Functional Family Therapy, they studied only fidelity to dosage, finding that 
clients at CII seemed to have received the model’s expected dosage of treatment. For Trauma- 
Focused Cognitive Behavioral Therapy (TF-CBT), the research team evaluated fidelity to the 
treatment model and dosage, finding that clients received close to the model’s expected dosage 
of treatment. However, in terms of fidelity to the treatment model, the team found that therapists 
on average did not deliver all of TF-CBT’s required treatment components. Clients receiving 
the average dosage of TF-CBT at CII had at least a 50 percent chance of receiving half of the 
required components. This finding is consistent with findings from similar TF-CBT fidelity 
studies. Notably, TF-CBT therapists at CII delivered the trauma narrative component at a rate 
higher than the average rate suggested by previous studies. These challenges underscore the 
need for effective tools to support fidelity. This study found that the therapist self-report tool, 
the Brief Practice Checklist, may be an effective, low-cost tool for monitoring fidelity, albeit 
with some limitations. 


Changes at CII 

This evaluation is based on data gathered over a period of approximately two years (2012 and 
2013), which was a period of tremendous change at CII. Conducting an assessment of the 
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Integrated Service Model while it was under development was challenging and limited the 
scope of the findings. During the study period, CII was in the process of scaling up its Integrat- 
ed Service Model, working to improve its data systems and how it measures outcomes, stream- 
lining its client enrollment and assessment process, and restructuring management. Additional- 
ly, during the study period, CII increased its operating budget as well as the number of clients it 
served and somewhat modified specific programs and services. In light of these many changes, 
the findings from this study should be interpreted with some caution. In retrospect, a study 
focused on CII’s development and scaling up of its Integrated Service Model may have been 
more appropriate than an implementation study. 


Lessons Learned and Next Steps 

Despite the many challenges, the study did produce some lessons. As policymakers, practition- 
ers, and researchers in the child welfare field work to improve services available through the 
child welfare system, CII and its experience developing and implementing its Integrated Service 
model as well as delivering evidence-based practices offer important lessons. How CII over- 
came systemic and other challenges and creatively wove together public and private funding 
streams to integrate its services could be infonnative not only to similar multiservice organiza- 
tions but to all those in the child welfare field looking for the best ways to serve children 
through an often fragmented child welfare system. 

Those interested in evidence-based practices may find the findings from the fidelity 
study of Trauma-Focused Cognitive Behavioral Therapy useful. These findings suggest that 
while self-monitoring tools may be effective and low cost, alone they are not enough to ensure 
fidelity. One area for further research could be investigating how to cost-effectively combine 
self-reporting tools and observational methods to support fidelity. 
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Appendix A 


Data Analysis Limitations 




This Appendix describes various challenges MDRC researchers encountered during the data 
analysis for this report. MDRC researchers began laying the groundwork for analysis in 2012, 
and over the years a number of issues arose which complicated the data analysis. 

The management information system that Children Institute, Inc. (CII), was using at the 
time of the study was designed not for research but for billing and reporting purposes. As a 
result, how CII categorized its services in the system did not align well with the field-relevant 
research questions and thus the service categories designed by the research team. The team did 
not fully understand the issue until after it had processed the first data files. The challenges 
MDRC encountered in the analysis fall into four overarching categories: 

• Integrated Service Model 

• Evidence-based practices 

• Outcome data 

• Staffing data 


Challenges Related to the Analysis of the Integrated 
Service Model 

CII’s Integrated Service Model is designed to address the holistic needs of the client and client’s 
family. As such, CII provides a variety of services to meet those needs, customized to each 
client and family. To analyze the implementation of the Integrated Service Model, the research 
team and CII staff organized CII’s services into three categories: clinical services, family 
support, and youth development. (See Chapters 2 and 4.) However, the management infor- 
mation system CII used to document its services and fulfill its contractual requirements was 
created primarily for reporting and billing purposes, and therefore the data were not already 
organized into those three categories. In order to fit the data fit into these categories, CII staff 
had to create a crosswalk between CII’s billing codes and the three service categories. 

As would be expected in any analysis of data gathered for one purpose and used for an- 
other, there were complications. 

• Multiple data systems. During the study period, CII maintained multiple da- 
ta systems. One system (TIER) captured the clinical service records and 
some family support and youth development activities, while a separate sys- 
tem (MINERVA) captured infonnation about participation in some youth 
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development activities. 1 There was not a universal client identifier that cut 
across the multiple data systems and clients could have multiple identifiers 
within the same system. Joining the clinical services data to the community 
services data was thus challenging and resulted in a couple of issues. 

o CII staff had to create a new identifier to join systems using a combina- 
tion of data points including birth date. There are likely instances where 
one individual still has multiple identifiers in the data set used in the 
analysis because of cases in which staff could not match records in one 
system to records in the other system (for example, if a client’s birth date 
was missing or entered incorrectly in one system). This issue could have 
led to the over-reporting of the total number of clients or the underreport- 
ing of the number of clients receiving multiple services. 

o The data systems did not connect the records of children to those of their 
parents or guardians. For example, a child may have received clinical 
services (documented in TIER) and his guardian may have received fi- 
nancial counseling (documented in MINERVA), but the research team 
could not tell from the data if the records were related or separate. In 
some cases, family support services were associated with the child’s rec- 
ord and in others they were associated with the parent’s or guardian’s 
record. This issue may have resulted in a misrepresentation of the clients’ 
age distribution, particularly in Table 4.5. 

• Differences in the types of data collected in each system. TIER and 
MINERVA gathered different information, making it difficult to analyze the 
merged data. For example, MINERVA included data primarily on activities 
for which only attendance was captured; in other words, the number of ses- 
sions attended was captured, but not the length of each session. TIER, in con- 
trast, included more detailed information about service time. The differences 
between the data in the two systems ultimately limited the analysis since it 
was not possible to analyze the client flow through all services without ser- 
vice dates for the activities recorded in MINERVA. 

• Service categories. Classifying CII services into clinical services, family 
support, and youth development categories was complex. While the research 
team described TIER as the “clinical services” system, it included records 


'in spring 2014, Cll adopted a new management information system that replaced the systems from which 
the research team pulled the 2012 and 2013 data. 
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from some youth development and family support services. Since procedure 
codes determined whether a service was classified as family support or clini- 
cal service, it was impossible to separate family support services that were 
packaged with a clinical service from those that were not. Thus while the 
service categories made understanding the analysis easier, it also limited the 
analysis in other ways. 

• Calculation of service time. TIER captured the allowable billing time for 
each service, not the actual amount of time a clinician or other staff person 
spent with a client. Allowable billing time could have included travel time to 
a client’s home or school and the time required to properly document a ses- 
sion or contact with the client. While this approach was necessary for billing 
purposes, it made calculating the face-to-face service time difficult. 

Calculating clients’ actual service time for group therapy from the available 
data was doubly complicated. TIER populated this field for each participant 
by dividing the group session time (including travel and documentation) by 
the number of participants in the group. As a result, the reported service 
times and calculated averages for individual clients were lower than the actu- 
al length of the group session. 

MINVERVA, on the other hand, did not include service time but only at- 
tendance records. The research team assigned session time to each activity 
based on information from CII staff about the length of each activity. For 
example, according to CII staff, acting and film production classes were 
assigned one hour and soccer programs were assigned two hours. As a re- 
sult, the service time data in TIER and MINVERVA were not directly 
comparable. 

In addition to these challenges, the research team also decided to exclude some data 
from the analysis. 

• Excluding one-day events. Youth development activities are wide rang- 
ing, from back-to-school nights, field trips, and holiday celebrations to tax 
preparation days, organized sports teams, and art classes. Youth develop- 
ment activities could be reoccurring activities or one-time events. The 
MDRC research team chose to exclude one-day events from the analysis. 

While one-day events may be central to CII’s involvement in the communi- 
ty, they were less central to the theory of the Integrated Service Model. 
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• Limiting analysis to clients who enrolled during a particular period. 

The Integrated Service Model calls for tailoring services to the needs of the 
individual and family and therefore CII sometimes offered services concur- 
rently, and other times consecutively; for example, it may take a client a 
year or more to transition from clinical to youth development services. To 
account for cases such as these, MDRC researchers combined 2012 and 
2013 data and analyzed service receipt across the two years. The research 
team limited the analysis to those clients who enrolled in CII services from 
January 1, 2012, through June 30, 2013; the team analyzed their records for 
the full two years. 


Challenges Related to Analysis of Evidence-Based Practices 

Data about the use of evidence-based practices is a subset of the overall set of data received 
from CII. Analysis of this data had its own difficulties. Only certain staff members were trained 
to administer evidence-based practices. For billing purposes, all records in TIER associated with 
a client receiving Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) or Functional 
Family Therapy (FFT) were coded for that therapy, including records for related services that 
are not considered therapy. For example, a FFT client could have received medication man- 
agement, which would have been coded as FFT though in fact it was not part of the therapy. 

To account for these situations, the MDRC research team limited its analysis only to 
those records with procedure codes that should have been attributed to the therapy. Further, to 
minimize the influence of data entry errors made by staff, the analysis included only records 
associated with staff trained in the specific evidence-based practice. This approach eliminated 
data entry errors that would have resulted in over counting. 


Challenges Related to Analysis of Outcome Data 

One goal of the study was to assess the outcomes of certain treatments. Data limitations, 
however, prevented the research team from conducting such analyses. Results of pre- and post- 
assessments were stored separate from the clinical services data. MDRC researchers received 
outcome data for clients engaged in five evidence-based practices in 2012 and 2013: TF-CBT, 
FFT, Cognitive Behavioral Intervention for Trauma in Schools, Incredible Years, and Manag- 
ing and Adapting Practice. Only a fraction of clients served in those years had any outcome 
records and many with pre-test scores were missing post-test scores. As a result, the research 
team did not have the required number of pre- and post-test score pairs needed for reliable 
analysis. For example, among clients starting TF-CBT in 2012 and 2013, fewer than one-third 


86 



had parent or guardian pre- and post-test pairs and only about one-fifth had pre- and post-test 
scores for the Youth Outcome Questionnaire. 


Challenges Related to Analysis of Staffing Data 

Data about staff training were also saved separate from the other data. Since CII stored training 
infomiation in different ways depending on the program or practice, some staff had incomplete 
data. The MDRC research team made the decision to only use staff training records that 
included a date when staff completed the training, since that indicated the training had been 
documented in the system. This decision is most applicable to Table 4.2. The analysis of staff 
training data was therefore limited to therapists and psychologists — those staff eligible to be 
trained in an evidence-based practice. 


Clinician Survey 

In spring 2013, the MDRC research team fielded a web-based survey to all current CII thera- 
pists and psychologists (69) and their supervisors (27). Of the 96 staff asked to complete the 
survey, 39 percent submitted responses. 

The survey, which included a combination of open- and closed-ended questions, ad- 
dressed the following topics: 

• Staff background. This series of questions asked about tenure at CII, as well 
as education and work background. It also included questions about their su- 
pervisory roles and movement across different positions within CII. 

• General Work. These questions asked about staff workload and opinions 
about documentation, assessments, outcomes, and referrals to youth devel- 
opment services. 

• Evidence-Based Treatments. This series of questions asked specifically 
about delivering evidence-based treatments or practices and the benefits and 
challenges of delivery. 

• Group Supervision for Evidence-Based Treatments. These questions 
were specifically about the operation of the group supervision component as- 
sociated with evidence-based practices and were intended to gauge the utility 
of these meetings. 

• Individual Supervision and Support. These questions asked about individ- 
ual supervision and other supports available to staff. 
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• CII Structure. These questions asked about overarching CII philosophies 
and the Integrated Service Model. 

The survey was designed to be anonymous, and thus the amount of infonnation collected about 
staff was limited, making it difficult to determine if the respondents were representative of all 
CII therapists or psychologists and their supervisors. Table A.l presents some descriptive 
infonnation about the respondents. 


Table A.l 

Characteristics of Clinician Survey Respondents 


Characteristic 

Therapists/ 

Psychologists 

Supervisors 

Average tenure at Cll (years) 

4 

16 

Held other positions at CII (%) 

29 

75 

Ever interned at CII (%) 

24 

13 

Licensed or registered Master's level social worker, 
marriage and family therapist, or psychologist 

33 

75 

Sample size 

21 

16 


SOURCE: MDRC calculations from the CII Clinician Survey. 
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